Base64 Decoding in Python 2.7

I am working with the following script that reads a csv file and decodes it. When I run it, I receive an Incorrect Padding error. When I take an individual entry and perform a str.decode('base64'), I get an appropriate result (no error). What could be the problem? Could some entries be corrupt and it's throwing the whole process off?

def get_sigpair_from_csv(csv_in, start=0, skip_to_tx=None, want_tx=):

    want_tx=set(want_tx)

    skip_entries = True

    with open(csv_in,'r') as f:

        for nr,line in enumerate(f):

            if nr<start:

                if nr%100000==0:

                    print "skip",nr,f.tell()

                continue

            if nr % 10000000 == 0:

                print "10m", nr

            try:

                # read data

                cols = line.split(";",1)

                tx = cols[0].strip()



                if skip_to_tx and tx==skip_to_tx:

                    skip_entries=False

                    # skip this entry - already in db

                    continue

                if skip_to_tx and skip_entries:

                    print "skiptx",nr, tx

                    continue

                if want_tx and tx not in want_tx:

                    continue



                scriptsig = cols[1].decode("base64")

                sig = scriptsig_to_ecdsa_sig(scriptsig)

                sig['tx'] = tx

                sig['nr'] = nr

                yield sig

            except ValueError, ve:

                #print tx,repr(ve)

                pass

            except Exception, e:

                print tx, repr(e)

Here is an example of the output:

879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd Error('Incorrect padding',)

However, when I do:

x = '879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd'

x.decode('base64')

It works

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

Python has builtin module called base64 docs.python.org/2/library/base64.html. You can simply use this module to encode base64.encodestring("some string") and base64.decodestring("some base64 encoded string") for encoding and decoding using base64

– sriharsha_bhat
Nov 20 '18 at 5:24

Unfortunately replacing cols[1].decode('base64') with the base64.decodestring() yields the same error

– GK89
Nov 20 '18 at 6:25

Base64 needs to be padded to even multiples of four characters. Show us the entry which fails. Maybe it's not base64 at all?

– tripleee
Nov 20 '18 at 7:29

Thanks for the sample. This looks like a hex digit, not base64. (The base64 repertoire includes all these characters, but also many more.)

– tripleee
Nov 20 '18 at 17:41

I'm guessing you left a newline at the end. Removing it will fix the immediate decoding error, but it's still garbage in, garbage out.

– tripleee
Nov 20 '18 at 17:45

|
show 2 more comments

def get_sigpair_from_csv(csv_in, start=0, skip_to_tx=None, want_tx=):

    want_tx=set(want_tx)

    skip_entries = True

    with open(csv_in,'r') as f:

        for nr,line in enumerate(f):

            if nr<start:

                if nr%100000==0:

                    print "skip",nr,f.tell()

                continue

            if nr % 10000000 == 0:

                print "10m", nr

            try:

                # read data

                cols = line.split(";",1)

                tx = cols[0].strip()



                if skip_to_tx and tx==skip_to_tx:

                    skip_entries=False

                    # skip this entry - already in db

                    continue

                if skip_to_tx and skip_entries:

                    print "skiptx",nr, tx

                    continue

                if want_tx and tx not in want_tx:

                    continue



                scriptsig = cols[1].decode("base64")

                sig = scriptsig_to_ecdsa_sig(scriptsig)

                sig['tx'] = tx

                sig['nr'] = nr

                yield sig

            except ValueError, ve:

                #print tx,repr(ve)

                pass

            except Exception, e:

                print tx, repr(e)

Here is an example of the output:

879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd Error('Incorrect padding',)

However, when I do:

x = '879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd'

x.decode('base64')

It works

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

Python has builtin module called base64 docs.python.org/2/library/base64.html. You can simply use this module to encode base64.encodestring("some string") and base64.decodestring("some base64 encoded string") for encoding and decoding using base64

– sriharsha_bhat
Nov 20 '18 at 5:24

Unfortunately replacing cols[1].decode('base64') with the base64.decodestring() yields the same error

– GK89
Nov 20 '18 at 6:25

Base64 needs to be padded to even multiples of four characters. Show us the entry which fails. Maybe it's not base64 at all?

– tripleee
Nov 20 '18 at 7:29

Thanks for the sample. This looks like a hex digit, not base64. (The base64 repertoire includes all these characters, but also many more.)

– tripleee
Nov 20 '18 at 17:41

I'm guessing you left a newline at the end. Removing it will fix the immediate decoding error, but it's still garbage in, garbage out.

– tripleee
Nov 20 '18 at 17:45

|
show 2 more comments

def get_sigpair_from_csv(csv_in, start=0, skip_to_tx=None, want_tx=):

    want_tx=set(want_tx)

    skip_entries = True

    with open(csv_in,'r') as f:

        for nr,line in enumerate(f):

            if nr<start:

                if nr%100000==0:

                    print "skip",nr,f.tell()

                continue

            if nr % 10000000 == 0:

                print "10m", nr

            try:

                # read data

                cols = line.split(";",1)

                tx = cols[0].strip()



                if skip_to_tx and tx==skip_to_tx:

                    skip_entries=False

                    # skip this entry - already in db

                    continue

                if skip_to_tx and skip_entries:

                    print "skiptx",nr, tx

                    continue

                if want_tx and tx not in want_tx:

                    continue



                scriptsig = cols[1].decode("base64")

                sig = scriptsig_to_ecdsa_sig(scriptsig)

                sig['tx'] = tx

                sig['nr'] = nr

                yield sig

            except ValueError, ve:

                #print tx,repr(ve)

                pass

            except Exception, e:

                print tx, repr(e)

Here is an example of the output:

879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd Error('Incorrect padding',)

However, when I do:

x = '879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd'

x.decode('base64')

It works

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

def get_sigpair_from_csv(csv_in, start=0, skip_to_tx=None, want_tx=):

    want_tx=set(want_tx)

    skip_entries = True

    with open(csv_in,'r') as f:

        for nr,line in enumerate(f):

            if nr<start:

                if nr%100000==0:

                    print "skip",nr,f.tell()

                continue

            if nr % 10000000 == 0:

                print "10m", nr

            try:

                # read data

                cols = line.split(";",1)

                tx = cols[0].strip()



                if skip_to_tx and tx==skip_to_tx:

                    skip_entries=False

                    # skip this entry - already in db

                    continue

                if skip_to_tx and skip_entries:

                    print "skiptx",nr, tx

                    continue

                if want_tx and tx not in want_tx:

                    continue



                scriptsig = cols[1].decode("base64")

                sig = scriptsig_to_ecdsa_sig(scriptsig)

                sig['tx'] = tx

                sig['nr'] = nr

                yield sig

            except ValueError, ve:

                #print tx,repr(ve)

                pass

            except Exception, e:

                print tx, repr(e)

Here is an example of the output:

879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd Error('Incorrect padding',)

However, when I do:

x = '879b068c75492e9d860763e843212d7aed2fb81ad3ee24592b48cdf5df624dcd'

x.decode('base64')

It works

python python-2.7 base64 decode

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

edited Nov 20 '18 at 14:34

asked Nov 20 '18 at 5:03

GK89

348217

asked Nov 20 '18 at 5:03

GK89

348217

asked Nov 20 '18 at 5:03

GK89

348217

Python has builtin module called base64 docs.python.org/2/library/base64.html. You can simply use this module to encode base64.encodestring("some string") and base64.decodestring("some base64 encoded string") for encoding and decoding using base64

– sriharsha_bhat
Nov 20 '18 at 5:24

Unfortunately replacing cols[1].decode('base64') with the base64.decodestring() yields the same error

– GK89
Nov 20 '18 at 6:25

Base64 needs to be padded to even multiples of four characters. Show us the entry which fails. Maybe it's not base64 at all?

– tripleee
Nov 20 '18 at 7:29

Thanks for the sample. This looks like a hex digit, not base64. (The base64 repertoire includes all these characters, but also many more.)

– tripleee
Nov 20 '18 at 17:41

I'm guessing you left a newline at the end. Removing it will fix the immediate decoding error, but it's still garbage in, garbage out.

– tripleee
Nov 20 '18 at 17:45

|
show 2 more comments

Python has builtin module called base64 docs.python.org/2/library/base64.html. You can simply use this module to encode base64.encodestring("some string") and base64.decodestring("some base64 encoded string") for encoding and decoding using base64

– sriharsha_bhat
Nov 20 '18 at 5:24

Unfortunately replacing cols[1].decode('base64') with the base64.decodestring() yields the same error

– GK89
Nov 20 '18 at 6:25

Base64 needs to be padded to even multiples of four characters. Show us the entry which fails. Maybe it's not base64 at all?

– tripleee
Nov 20 '18 at 7:29

Thanks for the sample. This looks like a hex digit, not base64. (The base64 repertoire includes all these characters, but also many more.)

– tripleee
Nov 20 '18 at 17:41

I'm guessing you left a newline at the end. Removing it will fix the immediate decoding error, but it's still garbage in, garbage out.

– tripleee
Nov 20 '18 at 17:45

Python has builtin module called base64 docs.python.org/2/library/base64.html. You can simply use this module to encode base64.encodestring("some string") and base64.decodestring("some base64 encoded string") for encoding and decoding using base64

– sriharsha_bhat
Nov 20 '18 at 5:24

Unfortunately replacing cols[1].decode('base64') with the base64.decodestring() yields the same error

– GK89
Nov 20 '18 at 6:25

Base64 needs to be padded to even multiples of four characters. Show us the entry which fails. Maybe it's not base64 at all?

– tripleee
Nov 20 '18 at 7:29

Thanks for the sample. This looks like a hex digit, not base64. (The base64 repertoire includes all these characters, but also many more.)

– tripleee
Nov 20 '18 at 17:41

I'm guessing you left a newline at the end. Removing it will fix the immediate decoding error, but it's still garbage in, garbage out.

– tripleee
Nov 20 '18 at 17:45

|
show 2 more comments

0

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53386515%2fbase64-decoding-in-python-2-7%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

0

active

oldest

votes

0

active

oldest

votes

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu