Mixing
Python delete row in file after reading it
up vote
-2
down vote
favorite
I python 2.7 I am reading data from file in while loop. When I successfully read row, I would like to delete this row from a file, but I dont know how to do it - Efficient way so i dont waste to much of CPU.
read = open("data.csv", 'r')
for row in read:
#code......
if send == True:
-->delete sent line from file, and continue to loop
python python-2.7
asked 2 days ago
TheUrban
73
add a comment |
up vote
-2
down vote
favorite
I python 2.7 I am reading data from file in while loop. When I successfully read row, I would like to delete this row from a file, but I dont know how to do it - Efficient way so i dont waste to much of CPU.
read = open("data.csv", 'r')
for row in read:
#code......
if send == True:
-->delete sent line from file, and continue to loop
python python-2.7
asked 2 days ago
TheUrban
73
add a comment |
up vote
-2
down vote
favorite
up vote
-2
down vote
favorite
I python 2.7 I am reading data from file in while loop. When I successfully read row, I would like to delete this row from a file, but I dont know how to do it - Efficient way so i dont waste to much of CPU.
read = open("data.csv", 'r')
for row in read:
#code......
if send == True:
-->delete sent line from file, and continue to loop
python python-2.7
asked 2 days ago
TheUrban
73
I python 2.7 I am reading data from file in while loop. When I successfully read row, I would like to delete this row from a file, but I dont know how to do it - Efficient way so i dont waste to much of CPU.
read = open("data.csv", 'r')
for row in read:
#code......
if send == True:
-->delete sent line from file, and continue to loop
python python-2.7
python python-2.7
asked 2 days ago
TheUrban
73
asked 2 days ago
TheUrban
73
asked 2 days ago
TheUrban
73
asked 2 days ago
TheUrban
73
asked 2 days ago
TheUrban
73
73
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
0
down vote
accepted
You shouldn't concern yourself about cpu usage when doing disk IO -- disk IO is very slow compared to almost any in-memory/cpu operation.
There are two strategies to deleting from the middle of a file:
writing all lines to keep to a secondary file, then renaming the secondary file to the original file name.
copy the rest (tail) of the file to the beginning of the line you want to delete and then truncating
xbytes off the end of the file (wherexis equal to the length of the line you want to remove.
Number 1 is usually preferred since it is easier and doesn't require any locking.
Mayank Porwal has given you most of strategy #1. Here is how you would implement strategy #2:
# open the file for both reading and writing in binary mode ('rb+')
with open('rmline.txt', 'rb+') as fp:
while 1:
pos = fp.tell() # remember the starting position of the next line to read
line = fp.readline()
if not line:
break # we reached the end of the file
if should_line_be_skipped(line): # only you know what to skip :-)
rest = fp.read() # read the rest of the file
fp.seek(pos) # go to the start position of the line to remove
fp.write(rest) # write the rest of the file over the line to be removed
fp.truncate() # truncates at current position (end of file - len(line))
fp.seek(pos) # return to where the next line is after deletion so we can continue the while loop
answered 2 days ago
thebjorn
12.5k44884
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
add a comment |
up vote
0
down vote
The best and the fastest option in my opinion would be to re-write the file elsewhere without the lines you wanted to delete.
Just have an appropriate negative condition to identify the lines which you want to keep.
with open("data.csv", 'r'):
with open("output.txt","w") as output:
for row in read:
#code......
if not send == True:
-->delete sent line from file, and continue to loop
output.write(line)
answered 2 days ago
Mayank Porwal
2,5341620
This:if ! send == True:is not valid Python code. Perhaps you meanif not send:.
– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
You shouldn't concern yourself about cpu usage when doing disk IO -- disk IO is very slow compared to almost any in-memory/cpu operation.
There are two strategies to deleting from the middle of a file:
writing all lines to keep to a secondary file, then renaming the secondary file to the original file name.
copy the rest (tail) of the file to the beginning of the line you want to delete and then truncating
xbytes off the end of the file (wherexis equal to the length of the line you want to remove.
Number 1 is usually preferred since it is easier and doesn't require any locking.
Mayank Porwal has given you most of strategy #1. Here is how you would implement strategy #2:
# open the file for both reading and writing in binary mode ('rb+')
with open('rmline.txt', 'rb+') as fp:
while 1:
pos = fp.tell() # remember the starting position of the next line to read
line = fp.readline()
if not line:
break # we reached the end of the file
if should_line_be_skipped(line): # only you know what to skip :-)
rest = fp.read() # read the rest of the file
fp.seek(pos) # go to the start position of the line to remove
fp.write(rest) # write the rest of the file over the line to be removed
fp.truncate() # truncates at current position (end of file - len(line))
fp.seek(pos) # return to where the next line is after deletion so we can continue the while loop
answered 2 days ago
thebjorn
12.5k44884
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
add a comment |
up vote
0
down vote
accepted
You shouldn't concern yourself about cpu usage when doing disk IO -- disk IO is very slow compared to almost any in-memory/cpu operation.
There are two strategies to deleting from the middle of a file:
writing all lines to keep to a secondary file, then renaming the secondary file to the original file name.
copy the rest (tail) of the file to the beginning of the line you want to delete and then truncating
xbytes off the end of the file (wherexis equal to the length of the line you want to remove.
Number 1 is usually preferred since it is easier and doesn't require any locking.
Mayank Porwal has given you most of strategy #1. Here is how you would implement strategy #2:
# open the file for both reading and writing in binary mode ('rb+')
with open('rmline.txt', 'rb+') as fp:
while 1:
pos = fp.tell() # remember the starting position of the next line to read
line = fp.readline()
if not line:
break # we reached the end of the file
if should_line_be_skipped(line): # only you know what to skip :-)
rest = fp.read() # read the rest of the file
fp.seek(pos) # go to the start position of the line to remove
fp.write(rest) # write the rest of the file over the line to be removed
fp.truncate() # truncates at current position (end of file - len(line))
fp.seek(pos) # return to where the next line is after deletion so we can continue the while loop
answered 2 days ago
thebjorn
12.5k44884
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
You shouldn't concern yourself about cpu usage when doing disk IO -- disk IO is very slow compared to almost any in-memory/cpu operation.
There are two strategies to deleting from the middle of a file:
writing all lines to keep to a secondary file, then renaming the secondary file to the original file name.
copy the rest (tail) of the file to the beginning of the line you want to delete and then truncating
xbytes off the end of the file (wherexis equal to the length of the line you want to remove.
Number 1 is usually preferred since it is easier and doesn't require any locking.
Mayank Porwal has given you most of strategy #1. Here is how you would implement strategy #2:
# open the file for both reading and writing in binary mode ('rb+')
with open('rmline.txt', 'rb+') as fp:
while 1:
pos = fp.tell() # remember the starting position of the next line to read
line = fp.readline()
if not line:
break # we reached the end of the file
if should_line_be_skipped(line): # only you know what to skip :-)
rest = fp.read() # read the rest of the file
fp.seek(pos) # go to the start position of the line to remove
fp.write(rest) # write the rest of the file over the line to be removed
fp.truncate() # truncates at current position (end of file - len(line))
fp.seek(pos) # return to where the next line is after deletion so we can continue the while loop
answered 2 days ago
thebjorn
12.5k44884
You shouldn't concern yourself about cpu usage when doing disk IO -- disk IO is very slow compared to almost any in-memory/cpu operation.
There are two strategies to deleting from the middle of a file:
writing all lines to keep to a secondary file, then renaming the secondary file to the original file name.
copy the rest (tail) of the file to the beginning of the line you want to delete and then truncating
xbytes off the end of the file (wherexis equal to the length of the line you want to remove.
Number 1 is usually preferred since it is easier and doesn't require any locking.
Mayank Porwal has given you most of strategy #1. Here is how you would implement strategy #2:
# open the file for both reading and writing in binary mode ('rb+')
with open('rmline.txt', 'rb+') as fp:
while 1:
pos = fp.tell() # remember the starting position of the next line to read
line = fp.readline()
if not line:
break # we reached the end of the file
if should_line_be_skipped(line): # only you know what to skip :-)
rest = fp.read() # read the rest of the file
fp.seek(pos) # go to the start position of the line to remove
fp.write(rest) # write the rest of the file over the line to be removed
fp.truncate() # truncates at current position (end of file - len(line))
fp.seek(pos) # return to where the next line is after deletion so we can continue the while loop
answered 2 days ago
thebjorn
12.5k44884
edited 2 days ago
answered 2 days ago
thebjorn
12.5k44884
answered 2 days ago
thebjorn
12.5k44884
answered 2 days ago
thebjorn
12.5k44884
12.5k44884
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
add a comment |
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
Yeah, thanks, I will do it this way. Because my problem was if for example, computer crash in the middle of reading large file, when it turns up again it needs to start from beginning. But this code will do as I want it, to store in loop. Thanks! :)
– TheUrban
2 days ago
add a comment |
up vote
0
down vote
The best and the fastest option in my opinion would be to re-write the file elsewhere without the lines you wanted to delete.
Just have an appropriate negative condition to identify the lines which you want to keep.
with open("data.csv", 'r'):
with open("output.txt","w") as output:
for row in read:
#code......
if not send == True:
-->delete sent line from file, and continue to loop
output.write(line)
answered 2 days ago
Mayank Porwal
2,5341620
This:if ! send == True:is not valid Python code. Perhaps you meanif not send:.
– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
add a comment |
up vote
0
down vote
The best and the fastest option in my opinion would be to re-write the file elsewhere without the lines you wanted to delete.
Just have an appropriate negative condition to identify the lines which you want to keep.
with open("data.csv", 'r'):
with open("output.txt","w") as output:
for row in read:
#code......
if not send == True:
-->delete sent line from file, and continue to loop
output.write(line)
answered 2 days ago
Mayank Porwal
2,5341620
This:if ! send == True:is not valid Python code. Perhaps you meanif not send:.
– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
add a comment |
up vote
0
down vote
up vote
0
down vote
The best and the fastest option in my opinion would be to re-write the file elsewhere without the lines you wanted to delete.
Just have an appropriate negative condition to identify the lines which you want to keep.
with open("data.csv", 'r'):
with open("output.txt","w") as output:
for row in read:
#code......
if not send == True:
-->delete sent line from file, and continue to loop
output.write(line)
answered 2 days ago
Mayank Porwal
2,5341620
The best and the fastest option in my opinion would be to re-write the file elsewhere without the lines you wanted to delete.
Just have an appropriate negative condition to identify the lines which you want to keep.
with open("data.csv", 'r'):
with open("output.txt","w") as output:
for row in read:
#code......
if not send == True:
-->delete sent line from file, and continue to loop
output.write(line)
answered 2 days ago
Mayank Porwal
2,5341620
edited 2 days ago
answered 2 days ago
Mayank Porwal
2,5341620
answered 2 days ago
Mayank Porwal
2,5341620
answered 2 days ago
Mayank Porwal
2,5341620
2,5341620
This:if ! send == True:is not valid Python code. Perhaps you meanif not send:.
– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
add a comment |
This:if ! send == True:is not valid Python code. Perhaps you meanif not send:.
– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
This:
if ! send == True: is not valid Python code. Perhaps you mean if not send:.– khelwood
2 days ago
This:
if ! send == True: is not valid Python code. Perhaps you mean if not send:.– khelwood
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
Yes, thanks for pointing that out. Fixed.
– Mayank Porwal
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
You'll want to open the from-file in binary mode, since you're opening the to-file in binary mode..
– thebjorn
2 days ago
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372874%2fpython-delete-row-in-file-after-reading-it%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
