Flink to Nifi the Magic Header was not present












2















I am trying to use this example to connect Nifi to Flink:



    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();

SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);

DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
@Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});

dataStream.print();
env.execute();


I am running Nifi as a standalone server with default properties, except these properties:



nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true


The call fails each time, with following log in Nifi:



[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection


Nifi version: 1.7.1, Flink version: 1.7.1










share|improve this question

























  • Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

    – daggett
    Jan 1 at 1:14











  • Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

    – Laabidi Raissi
    Jan 1 at 14:12
















2















I am trying to use this example to connect Nifi to Flink:



    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();

SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);

DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
@Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});

dataStream.print();
env.execute();


I am running Nifi as a standalone server with default properties, except these properties:



nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true


The call fails each time, with following log in Nifi:



[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection


Nifi version: 1.7.1, Flink version: 1.7.1










share|improve this question

























  • Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

    – daggett
    Jan 1 at 1:14











  • Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

    – Laabidi Raissi
    Jan 1 at 14:12














2












2








2








I am trying to use this example to connect Nifi to Flink:



    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();

SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);

DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
@Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});

dataStream.print();
env.execute();


I am running Nifi as a standalone server with default properties, except these properties:



nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true


The call fails each time, with following log in Nifi:



[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection


Nifi version: 1.7.1, Flink version: 1.7.1










share|improve this question
















I am trying to use this example to connect Nifi to Flink:



    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();

SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);

DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
@Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});

dataStream.print();
env.execute();


I am running Nifi as a standalone server with default properties, except these properties:



nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true


The call fails each time, with following log in Nifi:



[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener 
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection


Nifi version: 1.7.1, Flink version: 1.7.1







apache-flink apache-nifi






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Dec 31 '18 at 20:58







Laabidi Raissi

















asked Dec 31 '18 at 20:50









Laabidi RaissiLaabidi Raissi

2,69111326




2,69111326













  • Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

    – daggett
    Jan 1 at 1:14











  • Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

    – Laabidi Raissi
    Jan 1 at 14:12



















  • Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

    – daggett
    Jan 1 at 1:14











  • Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

    – Laabidi Raissi
    Jan 1 at 14:12

















Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14





Seems you try to connect flink to nifi by http... And nifi listening raw nifi protocol.

– daggett
Jan 1 at 1:14













Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12





Thanks for your hint, but I tried both transportProtocol(SiteToSiteTransportProtocol.RAW) and HTTP with the same result

– Laabidi Raissi
Jan 1 at 14:12












1 Answer
1






active

oldest

votes


















1














After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.



The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:



Unable to refresh Remote Group's peers due to Remote instance of NiFi 
is not configured to allow RAW Socket site-to-site communications


That's why I thought it was a port issue



So now with the default config of Nifi, this works as expected:



SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8080/nifi")
.portName("portNameAsInNifi")
.transportProtocol(SiteToSiteTransportProtocol.HTTP)
.requestBatchCount(1)
.buildConfig();





share|improve this answer

























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53991316%2fflink-to-nifi-the-magic-header-was-not-present%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.



    The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
    And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:



    Unable to refresh Remote Group's peers due to Remote instance of NiFi 
    is not configured to allow RAW Socket site-to-site communications


    That's why I thought it was a port issue



    So now with the default config of Nifi, this works as expected:



    SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
    .url("http://localhost:8080/nifi")
    .portName("portNameAsInNifi")
    .transportProtocol(SiteToSiteTransportProtocol.HTTP)
    .requestBatchCount(1)
    .buildConfig();





    share|improve this answer






























      1














      After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.



      The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
      And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:



      Unable to refresh Remote Group's peers due to Remote instance of NiFi 
      is not configured to allow RAW Socket site-to-site communications


      That's why I thought it was a port issue



      So now with the default config of Nifi, this works as expected:



      SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
      .url("http://localhost:8080/nifi")
      .portName("portNameAsInNifi")
      .transportProtocol(SiteToSiteTransportProtocol.HTTP)
      .requestBatchCount(1)
      .buildConfig();





      share|improve this answer




























        1












        1








        1







        After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.



        The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
        And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:



        Unable to refresh Remote Group's peers due to Remote instance of NiFi 
        is not configured to allow RAW Socket site-to-site communications


        That's why I thought it was a port issue



        So now with the default config of Nifi, this works as expected:



        SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
        .url("http://localhost:8080/nifi")
        .portName("portNameAsInNifi")
        .transportProtocol(SiteToSiteTransportProtocol.HTTP)
        .requestBatchCount(1)
        .buildConfig();





        share|improve this answer















        After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.



        The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
        And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:



        Unable to refresh Remote Group's peers due to Remote instance of NiFi 
        is not configured to allow RAW Socket site-to-site communications


        That's why I thought it was a port issue



        So now with the default config of Nifi, this works as expected:



        SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
        .url("http://localhost:8080/nifi")
        .portName("portNameAsInNifi")
        .transportProtocol(SiteToSiteTransportProtocol.HTTP)
        .requestBatchCount(1)
        .buildConfig();






        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Jan 1 at 15:12

























        answered Jan 1 at 15:07









        Laabidi RaissiLaabidi Raissi

        2,69111326




        2,69111326
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53991316%2fflink-to-nifi-the-magic-header-was-not-present%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            'app-layout' is not a known element: how to share Component with different Modules

            SQL update select statement

            android studio warns about leanback feature tag usage required on manifest while using Unity exported app?