Flink to Nifi the Magic Header was not present

I am trying to use this example to connect Nifi to Flink:

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



    SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8090/nifi")

            .portName("Data for Flink")

            .requestBatchCount(5)

            .buildConfig();



    SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);

    DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);



    DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {

        @Override

        public String map(NiFiDataPacket value) throws Exception {

            return new String(value.getContent(), Charset.defaultCharset());

        }

    });



    dataStream.print();

    env.execute();

I am running Nifi as a standalone server with default properties, except these properties:

nifi.remote.input.host=localhost

nifi.remote.input.secure=false

nifi.remote.input.socket.port=8090

nifi.remote.input.http.enabled=true

The call fails each time, with following log in Nifi:

[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 

Unable to communicate with remote instance null due to

org.apache.nifi.remote.exception.HandshakeException: Handshake 

with nifi://localhost:61680 failed because the Magic Header 

was not present; closing connection

Nifi version: 1.7.1, Flink version: 1.7.1

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14

Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12

add a comment |

I am trying to use this example to connect Nifi to Flink:

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



    SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8090/nifi")

            .portName("Data for Flink")

            .requestBatchCount(5)

            .buildConfig();



    SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);

    DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);



    DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {

        @Override

        public String map(NiFiDataPacket value) throws Exception {

            return new String(value.getContent(), Charset.defaultCharset());

        }

    });



    dataStream.print();

    env.execute();

I am running Nifi as a standalone server with default properties, except these properties:

nifi.remote.input.host=localhost

nifi.remote.input.secure=false

nifi.remote.input.socket.port=8090

nifi.remote.input.http.enabled=true

The call fails each time, with following log in Nifi:

[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 

Unable to communicate with remote instance null due to

org.apache.nifi.remote.exception.HandshakeException: Handshake 

with nifi://localhost:61680 failed because the Magic Header 

was not present; closing connection

Nifi version: 1.7.1, Flink version: 1.7.1

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14

Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12

add a comment |

I am trying to use this example to connect Nifi to Flink:

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



    SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8090/nifi")

            .portName("Data for Flink")

            .requestBatchCount(5)

            .buildConfig();



    SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);

    DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);



    DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {

        @Override

        public String map(NiFiDataPacket value) throws Exception {

            return new String(value.getContent(), Charset.defaultCharset());

        }

    });



    dataStream.print();

    env.execute();

I am running Nifi as a standalone server with default properties, except these properties:

nifi.remote.input.host=localhost

nifi.remote.input.secure=false

nifi.remote.input.socket.port=8090

nifi.remote.input.http.enabled=true

The call fails each time, with following log in Nifi:

[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 

Unable to communicate with remote instance null due to

org.apache.nifi.remote.exception.HandshakeException: Handshake 

with nifi://localhost:61680 failed because the Magic Header 

was not present; closing connection

Nifi version: 1.7.1, Flink version: 1.7.1

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

I am trying to use this example to connect Nifi to Flink:

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();



    SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8090/nifi")

            .portName("Data for Flink")

            .requestBatchCount(5)

            .buildConfig();



    SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);

    DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);



    DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {

        @Override

        public String map(NiFiDataPacket value) throws Exception {

            return new String(value.getContent(), Charset.defaultCharset());

        }

    });



    dataStream.print();

    env.execute();

I am running Nifi as a standalone server with default properties, except these properties:

nifi.remote.input.host=localhost

nifi.remote.input.secure=false

nifi.remote.input.socket.port=8090

nifi.remote.input.http.enabled=true

The call fails each time, with following log in Nifi:

[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 

Unable to communicate with remote instance null due to

org.apache.nifi.remote.exception.HandshakeException: Handshake 

with nifi://localhost:61680 failed because the Magic Header 

was not present; closing connection

Nifi version: 1.7.1, Flink version: 1.7.1

apache-flink apache-nifi

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

edited Dec 31 '18 at 20:58

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

asked Dec 31 '18 at 20:50

Laabidi Raissi

2,69111326

Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14

Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12

add a comment |

Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14

Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12

Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14

Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12

add a comment |

1 Answer
1

active

oldest

votes

After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.

The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:

Unable to refresh Remote Group's peers due to Remote instance of NiFi 

is not configured to allow RAW Socket site-to-site communications

That's why I thought it was a port issue

So now with the default config of Nifi, this works as expected:

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8080/nifi")

            .portName("portNameAsInNifi")

            .transportProtocol(SiteToSiteTransportProtocol.HTTP)

            .requestBatchCount(1)

            .buildConfig();

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53991316%2fflink-to-nifi-the-magic-header-was-not-present%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Unable to refresh Remote Group's peers due to Remote instance of NiFi 

is not configured to allow RAW Socket site-to-site communications

That's why I thought it was a port issue

So now with the default config of Nifi, this works as expected:

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8080/nifi")

            .portName("portNameAsInNifi")

            .transportProtocol(SiteToSiteTransportProtocol.HTTP)

            .requestBatchCount(1)

            .buildConfig();

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

add a comment |

Unable to refresh Remote Group's peers due to Remote instance of NiFi 

is not configured to allow RAW Socket site-to-site communications

That's why I thought it was a port issue

So now with the default config of Nifi, this works as expected:

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8080/nifi")

            .portName("portNameAsInNifi")

            .transportProtocol(SiteToSiteTransportProtocol.HTTP)

            .requestBatchCount(1)

            .buildConfig();

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

add a comment |

Unable to refresh Remote Group's peers due to Remote instance of NiFi 

is not configured to allow RAW Socket site-to-site communications

That's why I thought it was a port issue

So now with the default config of Nifi, this works as expected:

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8080/nifi")

            .portName("portNameAsInNifi")

            .transportProtocol(SiteToSiteTransportProtocol.HTTP)

            .requestBatchCount(1)

            .buildConfig();

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

Unable to refresh Remote Group's peers due to Remote instance of NiFi 

is not configured to allow RAW Socket site-to-site communications

That's why I thought it was a port issue

So now with the default config of Nifi, this works as expected:

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

            .url("http://localhost:8080/nifi")

            .portName("portNameAsInNifi")

            .transportProtocol(SiteToSiteTransportProtocol.HTTP)

            .requestBatchCount(1)

            .buildConfig();

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

edited Jan 1 at 15:12

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

answered Jan 1 at 15:07

Laabidi Raissi

2,69111326

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu