Mixing
remove all lines in a file containing a string from another file
1
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
edited Nov 21 '18 at 7:23
Inian
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
|
show 3 more comments
1
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
edited Nov 21 '18 at 7:23
Inian
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08
|
show 3 more comments
1
1
1
3
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
edited Nov 21 '18 at 7:23
Inian
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
awk grep match carriage-return
edited Nov 21 '18 at 7:23
Inian
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
edited Nov 21 '18 at 7:23
Inian
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
edited Nov 21 '18 at 7:23
Inian
39.4k63971
edited Nov 21 '18 at 7:23
Inian
39.4k63971
edited Nov 21 '18 at 7:23
Inian
39.4k63971
39.4k63971
asked Nov 21 '18 at 2:33
NKNNKN
305
asked Nov 21 '18 at 2:33
NKNNKN
305
asked Nov 21 '18 at 2:33
NKNNKN
305
305
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08
|
show 3 more comments
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08
|
show 3 more comments
1 Answer
1
active
oldest
votes
3
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
1
@NKN, also try to check by doingcat -v Input_fileif you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?
– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404530%2fremove-all-lines-in-a-file-containing-a-string-from-another-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
3
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
1
@NKN, also try to check by doingcat -v Input_fileif you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?
– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
add a comment |
3
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
1
@NKN, also try to check by doingcat -v Input_fileif you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?
– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
add a comment |
3
3
3
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
edited Nov 21 '18 at 3:43
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
answered Nov 21 '18 at 2:51
RavinderSingh13RavinderSingh13
27.3k41538
27.3k41538
1
@NKN, also try to check by doingcat -v Input_fileif you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?
– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
add a comment |
1
@NKN, also try to check by doingcat -v Input_fileif you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?
– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
1
1
@NKN, also try to check by doing
cat -v Input_file if you have carriage returns in your file? If yes then you could remove them by doing tr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?– RavinderSingh13
Nov 21 '18 at 3:20
@NKN, also try to check by doing
cat -v Input_file if you have carriage returns in your file? If yes then you could remove them by doing tr -d 'r' < Input_file > temp_file && mv temp_file Input_file, let me know on same?– RavinderSingh13
Nov 21 '18 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 '18 at 3:34
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404530%2fremove-all-lines-in-a-file-containing-a-string-from-another-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 '18 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 '18 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 '18 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 '18 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 '18 at 3:08