Mixing
Optimizing the replacement of parts of text documents
0
So I'm trying to read a bunch of .mcfunction files (basically .txt) and replace things that are contained inside of || with the corresponding number. But the way I'm doing it now causes huge memory usage (like 360mb --> 6gb), so I need help fixing the memory leak. This is where the problem is:
public void Parse() {
staticIDs sIDs = GameObject.Find("Main Camera").GetComponent<staticIDs>();
functions = Directory.GetFiles(@"downloads", "*.mcfunction", SearchOption.AllDirectories);
totalFunctions = functions.Length;
curFunction = 0;
foreach (string function in functions)
{
System.Threading.Thread.Sleep(100);
string text = File.ReadAllText(function);
if (text.Contains("|"))
{
Debug.Log("Things to parse");
goto contParse;
}
else {
Debug.Log("No things to parse");
goto finishParse;
}
contParse:
lines = File.ReadAllLines(function);
int lineNumber = 0;
foreach (string line in lines)
{
System.Threading.Thread.Sleep(100);
parts = line.Split('|');
temp = "";
int curLine = lineNumber + 1;
foreach (string part in parts)
{
System.Threading.Thread.Sleep(100);
int IDchecked = 0;
done = false;
tryAgain:
if (IDchecked < sIDs.IDnames.Capacity && sIDs.IDnames[IDchecked] == part)
{
temp = temp + sIDs.IDnumbers[IDchecked].ToString();
goto replaced;
}
else if (IDchecked <= sIDs.IDnames.Capacity)
{
IDchecked++;
goto tryAgain;
}
else if (IDchecked > sIDs.IDnames.Capacity)
{
temp = temp + part;
goto replaced;
}
else
{
goto replaced;
}
replaced:
Debug.Log(temp);
}
done = false;
newLines.Add(temp);
}
File.Delete(function);
File.Create(function).Dispose();
foreach (string line in newLines)
{
File.AppendAllText(function, line);
}
finishParse:
curFunction++;
if (curFunction > totalFunctions)
goto move;
}
move:
Debug.Log("got to moving");
Invoke("Move", 10f);
}
I would like to get the memory usage down to as little as possible.
c# performance unity3d replace
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
|
show 2 more comments
0
So I'm trying to read a bunch of .mcfunction files (basically .txt) and replace things that are contained inside of || with the corresponding number. But the way I'm doing it now causes huge memory usage (like 360mb --> 6gb), so I need help fixing the memory leak. This is where the problem is:
public void Parse() {
staticIDs sIDs = GameObject.Find("Main Camera").GetComponent<staticIDs>();
functions = Directory.GetFiles(@"downloads", "*.mcfunction", SearchOption.AllDirectories);
totalFunctions = functions.Length;
curFunction = 0;
foreach (string function in functions)
{
System.Threading.Thread.Sleep(100);
string text = File.ReadAllText(function);
if (text.Contains("|"))
{
Debug.Log("Things to parse");
goto contParse;
}
else {
Debug.Log("No things to parse");
goto finishParse;
}
contParse:
lines = File.ReadAllLines(function);
int lineNumber = 0;
foreach (string line in lines)
{
System.Threading.Thread.Sleep(100);
parts = line.Split('|');
temp = "";
int curLine = lineNumber + 1;
foreach (string part in parts)
{
System.Threading.Thread.Sleep(100);
int IDchecked = 0;
done = false;
tryAgain:
if (IDchecked < sIDs.IDnames.Capacity && sIDs.IDnames[IDchecked] == part)
{
temp = temp + sIDs.IDnumbers[IDchecked].ToString();
goto replaced;
}
else if (IDchecked <= sIDs.IDnames.Capacity)
{
IDchecked++;
goto tryAgain;
}
else if (IDchecked > sIDs.IDnames.Capacity)
{
temp = temp + part;
goto replaced;
}
else
{
goto replaced;
}
replaced:
Debug.Log(temp);
}
done = false;
newLines.Add(temp);
}
File.Delete(function);
File.Create(function).Dispose();
foreach (string line in newLines)
{
File.AppendAllText(function, line);
}
finishParse:
curFunction++;
if (curFunction > totalFunctions)
goto move;
}
move:
Debug.Log("got to moving");
Invoke("Move", 10f);
}
I would like to get the memory usage down to as little as possible.
c# performance unity3d replace
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from usinggoto.
– Jonathon Chase
Jan 3 at 1:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
Reopening file for every line should make your code slow enough already without sleep... Also that insanegotolook like some sort offorloop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)
– Alexei Levenkov
Jan 3 at 2:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32
|
show 2 more comments
0
0
0
So I'm trying to read a bunch of .mcfunction files (basically .txt) and replace things that are contained inside of || with the corresponding number. But the way I'm doing it now causes huge memory usage (like 360mb --> 6gb), so I need help fixing the memory leak. This is where the problem is:
public void Parse() {
staticIDs sIDs = GameObject.Find("Main Camera").GetComponent<staticIDs>();
functions = Directory.GetFiles(@"downloads", "*.mcfunction", SearchOption.AllDirectories);
totalFunctions = functions.Length;
curFunction = 0;
foreach (string function in functions)
{
System.Threading.Thread.Sleep(100);
string text = File.ReadAllText(function);
if (text.Contains("|"))
{
Debug.Log("Things to parse");
goto contParse;
}
else {
Debug.Log("No things to parse");
goto finishParse;
}
contParse:
lines = File.ReadAllLines(function);
int lineNumber = 0;
foreach (string line in lines)
{
System.Threading.Thread.Sleep(100);
parts = line.Split('|');
temp = "";
int curLine = lineNumber + 1;
foreach (string part in parts)
{
System.Threading.Thread.Sleep(100);
int IDchecked = 0;
done = false;
tryAgain:
if (IDchecked < sIDs.IDnames.Capacity && sIDs.IDnames[IDchecked] == part)
{
temp = temp + sIDs.IDnumbers[IDchecked].ToString();
goto replaced;
}
else if (IDchecked <= sIDs.IDnames.Capacity)
{
IDchecked++;
goto tryAgain;
}
else if (IDchecked > sIDs.IDnames.Capacity)
{
temp = temp + part;
goto replaced;
}
else
{
goto replaced;
}
replaced:
Debug.Log(temp);
}
done = false;
newLines.Add(temp);
}
File.Delete(function);
File.Create(function).Dispose();
foreach (string line in newLines)
{
File.AppendAllText(function, line);
}
finishParse:
curFunction++;
if (curFunction > totalFunctions)
goto move;
}
move:
Debug.Log("got to moving");
Invoke("Move", 10f);
}
I would like to get the memory usage down to as little as possible.
c# performance unity3d replace
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
So I'm trying to read a bunch of .mcfunction files (basically .txt) and replace things that are contained inside of || with the corresponding number. But the way I'm doing it now causes huge memory usage (like 360mb --> 6gb), so I need help fixing the memory leak. This is where the problem is:
public void Parse() {
staticIDs sIDs = GameObject.Find("Main Camera").GetComponent<staticIDs>();
functions = Directory.GetFiles(@"downloads", "*.mcfunction", SearchOption.AllDirectories);
totalFunctions = functions.Length;
curFunction = 0;
foreach (string function in functions)
{
System.Threading.Thread.Sleep(100);
string text = File.ReadAllText(function);
if (text.Contains("|"))
{
Debug.Log("Things to parse");
goto contParse;
}
else {
Debug.Log("No things to parse");
goto finishParse;
}
contParse:
lines = File.ReadAllLines(function);
int lineNumber = 0;
foreach (string line in lines)
{
System.Threading.Thread.Sleep(100);
parts = line.Split('|');
temp = "";
int curLine = lineNumber + 1;
foreach (string part in parts)
{
System.Threading.Thread.Sleep(100);
int IDchecked = 0;
done = false;
tryAgain:
if (IDchecked < sIDs.IDnames.Capacity && sIDs.IDnames[IDchecked] == part)
{
temp = temp + sIDs.IDnumbers[IDchecked].ToString();
goto replaced;
}
else if (IDchecked <= sIDs.IDnames.Capacity)
{
IDchecked++;
goto tryAgain;
}
else if (IDchecked > sIDs.IDnames.Capacity)
{
temp = temp + part;
goto replaced;
}
else
{
goto replaced;
}
replaced:
Debug.Log(temp);
}
done = false;
newLines.Add(temp);
}
File.Delete(function);
File.Create(function).Dispose();
foreach (string line in newLines)
{
File.AppendAllText(function, line);
}
finishParse:
curFunction++;
if (curFunction > totalFunctions)
goto move;
}
move:
Debug.Log("got to moving");
Invoke("Move", 10f);
}
I would like to get the memory usage down to as little as possible.
c# performance unity3d replace
c# performance unity3d replace
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
asked Jan 3 at 1:42
The Nuclear NexusThe Nuclear Nexus
1
1
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from usinggoto.
– Jonathon Chase
Jan 3 at 1:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
Reopening file for every line should make your code slow enough already without sleep... Also that insanegotolook like some sort offorloop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)
– Alexei Levenkov
Jan 3 at 2:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32
|
show 2 more comments
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from usinggoto.
– Jonathon Chase
Jan 3 at 1:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
Reopening file for every line should make your code slow enough already without sleep... Also that insanegotolook like some sort offorloop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)
– Alexei Levenkov
Jan 3 at 2:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from using
goto.– Jonathon Chase
Jan 3 at 1:48
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from using
goto.– Jonathon Chase
Jan 3 at 1:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
Reopening file for every line should make your code slow enough already without sleep... Also that insane
goto look like some sort of for loop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)– Alexei Levenkov
Jan 3 at 2:32
Reopening file for every line should make your code slow enough already without sleep... Also that insane
goto look like some sort of for loop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)– Alexei Levenkov
Jan 3 at 2:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32
|
show 2 more comments
1 Answer
1
active
oldest
votes
0
The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead
instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)
edited Jan 3 at 21:59
Glurth
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015350%2foptimizing-the-replacement-of-parts-of-text-documents%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead
instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)
edited Jan 3 at 21:59
Glurth
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
add a comment |
0
The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead
instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)
edited Jan 3 at 21:59
Glurth
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
add a comment |
0
0
0
The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead
instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)
edited Jan 3 at 21:59
Glurth
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
The memory leak is likely caused by string concatenation. string + string creates a brand new (immutable) string. Avoid those, and use a StringBuilder instead
instead of doing temp = temp + something, make temp StringBuilderer and call temp.Append(something)
edited Jan 3 at 21:59
Glurth
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
edited Jan 3 at 21:59
Glurth
215112
edited Jan 3 at 21:59
Glurth
215112
edited Jan 3 at 21:59
Glurth
215112
215112
answered Jan 3 at 12:32
zambarizambari
2,2331316
answered Jan 3 at 12:32
zambarizambari
2,2331316
answered Jan 3 at 12:32
zambarizambari
2,2331316
2,2331316
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
add a comment |
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
I can't seem to find string.Append? Maybe its called something else?
– The Nuclear Nexus
Jan 3 at 15:20
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
@TheNuclearNexus "StringBuilder" is a separate class from "string". Its functions, take and return "string"s.
– Glurth
Jan 3 at 17:57
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54015350%2foptimizing-the-replacement-of-parts-of-text-documents%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What's the total size of all the files you're trying to modify? Unrelated, you may want to refactor away from using
goto.– Jonathon Chase
Jan 3 at 1:48
What are all the sleeps for? I also agree with getting rid of the gotos. You might also consider if you've already read the whole file it's probably better to split that text into lines rather than read the whole file again. Same with writing a line at a time. Joint the lines and write it all at once.
– Retired Ninja
Jan 3 at 1:50
The sleeps where me trying to slow it down. The file size is relativly small but there are 100+ files. And I do split the text into lines but how do I write them all at once whilst maintaining the proper fortmatting?
– The Nuclear Nexus
Jan 3 at 2:00
Reopening file for every line should make your code slow enough already without sleep... Also that insane
gotolook like some sort offorloop … decompiled. Consider rewriting it by hand so it looks real. After that someone likely can give you advice on actual code. (In mean time check out stackoverflow.com/questions/7351031/… which likely will be better than … you are doing)– Alexei Levenkov
Jan 3 at 2:32
getting rid of those gotos and swapping to a for loop really helped ty
– The Nuclear Nexus
Jan 3 at 3:32