Mixing
speech to text in microsoft botframework using c#
I created a bot using microsoft botframework in c#. Now I am using speech recognition, in that I want to know one thing, can we control microphone on/off instead of clicking it for every question in webchat if it is not possible then please tell me how can we give speech as input and get output response in text from bot. Now it is giving text as response when I give text as input and speech as response when I give speech as input but I want it should respond in text when I give speech as input.
Thanks in advance.
c#
botframework speech-to-text asked Nov 20 '18 at 7:35
MounikaMounika
153
add a comment |
I created a bot using microsoft botframework in c#. Now I am using speech recognition, in that I want to know one thing, can we control microphone on/off instead of clicking it for every question in webchat if it is not possible then please tell me how can we give speech as input and get output response in text from bot. Now it is giving text as response when I give text as input and speech as response when I give speech as input but I want it should respond in text when I give speech as input.
Thanks in advance.
c#
botframework speech-to-text asked Nov 20 '18 at 7:35
MounikaMounika
153
1
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35
add a comment |
I created a bot using microsoft botframework in c#. Now I am using speech recognition, in that I want to know one thing, can we control microphone on/off instead of clicking it for every question in webchat if it is not possible then please tell me how can we give speech as input and get output response in text from bot. Now it is giving text as response when I give text as input and speech as response when I give speech as input but I want it should respond in text when I give speech as input.
Thanks in advance.
c#
botframework speech-to-text asked Nov 20 '18 at 7:35
MounikaMounika
153
I created a bot using microsoft botframework in c#. Now I am using speech recognition, in that I want to know one thing, can we control microphone on/off instead of clicking it for every question in webchat if it is not possible then please tell me how can we give speech as input and get output response in text from bot. Now it is giving text as response when I give text as input and speech as response when I give speech as input but I want it should respond in text when I give speech as input.
Thanks in advance.
c#
botframework speech-to-text c#
botframework speech-to-text asked Nov 20 '18 at 7:35
MounikaMounika
153
asked Nov 20 '18 at 7:35
MounikaMounika
153
asked Nov 20 '18 at 7:35
MounikaMounika
153
asked Nov 20 '18 at 7:35
MounikaMounika
153
asked Nov 20 '18 at 7:35
MounikaMounika
153
153
1
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35
add a comment |
1
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35
1
1
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35
add a comment |
2 Answers
2
active
oldest
votes
This depends on the SDK version you are using.
For v3, there are no out-of-the-box solutions you can use. As mentioned in this discussion, you may need to code a custom solution yourself to meet your needs. However, this discussion suggests that there may already be a solution you could possibly adapt to your needs.
For v4, there is no out-of-the-box solution at this time, but it is under consideration with some development already performed. Read that discussion here.
Hope of help!
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
add a comment |
If you want the mic to turn on after the bot speaks to you, you need to set the inputHint on the activity to ExpectingInput. If you always want that and you don't send multiple messages at a time, then you can set it with an ActivityMapper (v3) or a Middleware (v4). Just a note that with v3 the ExpectingInput breaks on ios. I'm not sure about v4 though. Below is an example middleware (v4) I have that sets the inputHint to ExpectingInput if the text or attachment ends with a "?".
public class TextToSpeechMiddleware : IMiddleware
{
public Task OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken = default(CancellationToken))
{
turnContext.OnSendActivities(OnSendActivities);
turnContext.OnUpdateActivity(OnUpdateActivity);
return next(cancellationToken);
}
private Task<ResourceResponse> OnUpdateActivity(ITurnContext turnContext, Activity activity, Func<Task<ResourceResponse>> next)
{
ConvertTextToSpeech(activity);
return next();
}
private Task<ResourceResponse> OnSendActivities(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse>> next)
{
foreach (Activity currentActivity in activities.Where(a => a.Type == ActivityTypes.Message))
{
ConvertTextToSpeech(currentActivity);
}
return next();
}
private void ConvertTextToSpeech(Activity message)
{
Activity initialMessage = message;
try
{
if (message.Type == ActivityTypes.Message)
{
if (string.IsNullOrEmpty(message.Speak))
{
if (string.IsNullOrEmpty(message.Text))
{
if (message.Attachments[0].Content is HeroCard attachment)
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(attachment.Text);
}
}
else
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(message.Text);
}
message.Speak = message.Speak.Trim();
if (ignoreList.Where(i => message.Speak.ToLower().StartsWith(i.ToLower())).Count() != 0)
{
message.Speak = null;
ignoredSpeak = true;
}
}
else if (string.IsNullOrWhiteSpace(message.Speak))
{
message.Speak = null;
}
if ((!string.IsNullOrEmpty(message.Speak) && (message.Speak.EndsWith("?") || message.Speak.StartsWith("Is this correct?")))
|| (!string.IsNullOrEmpty(message.Text) && message.Text.EndsWith("?")))
{
message.InputHint = InputHints.ExpectingInput;
}
// IOs won't work with expecting input
if (message.Recipient.Name.EndsWith(":ios"))
{
message.InputHint = InputHints.AcceptingInput;
}
}
}
catch (Exception)
{
message = initialMessage;
}
}
}
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388242%2fspeech-to-text-in-microsoft-botframework-using-c-sharp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
This depends on the SDK version you are using.
For v3, there are no out-of-the-box solutions you can use. As mentioned in this discussion, you may need to code a custom solution yourself to meet your needs. However, this discussion suggests that there may already be a solution you could possibly adapt to your needs.
For v4, there is no out-of-the-box solution at this time, but it is under consideration with some development already performed. Read that discussion here.
Hope of help!
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
add a comment |
This depends on the SDK version you are using.
For v3, there are no out-of-the-box solutions you can use. As mentioned in this discussion, you may need to code a custom solution yourself to meet your needs. However, this discussion suggests that there may already be a solution you could possibly adapt to your needs.
For v4, there is no out-of-the-box solution at this time, but it is under consideration with some development already performed. Read that discussion here.
Hope of help!
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
add a comment |
This depends on the SDK version you are using.
For v3, there are no out-of-the-box solutions you can use. As mentioned in this discussion, you may need to code a custom solution yourself to meet your needs. However, this discussion suggests that there may already be a solution you could possibly adapt to your needs.
For v4, there is no out-of-the-box solution at this time, but it is under consideration with some development already performed. Read that discussion here.
Hope of help!
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
This depends on the SDK version you are using.
For v3, there are no out-of-the-box solutions you can use. As mentioned in this discussion, you may need to code a custom solution yourself to meet your needs. However, this discussion suggests that there may already be a solution you could possibly adapt to your needs.
For v4, there is no out-of-the-box solution at this time, but it is under consideration with some development already performed. Read that discussion here.
Hope of help!
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
answered Nov 20 '18 at 19:34
Steven KanbergSteven Kanberg
7011618
7011618
add a comment |
add a comment |
If you want the mic to turn on after the bot speaks to you, you need to set the inputHint on the activity to ExpectingInput. If you always want that and you don't send multiple messages at a time, then you can set it with an ActivityMapper (v3) or a Middleware (v4). Just a note that with v3 the ExpectingInput breaks on ios. I'm not sure about v4 though. Below is an example middleware (v4) I have that sets the inputHint to ExpectingInput if the text or attachment ends with a "?".
public class TextToSpeechMiddleware : IMiddleware
{
public Task OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken = default(CancellationToken))
{
turnContext.OnSendActivities(OnSendActivities);
turnContext.OnUpdateActivity(OnUpdateActivity);
return next(cancellationToken);
}
private Task<ResourceResponse> OnUpdateActivity(ITurnContext turnContext, Activity activity, Func<Task<ResourceResponse>> next)
{
ConvertTextToSpeech(activity);
return next();
}
private Task<ResourceResponse> OnSendActivities(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse>> next)
{
foreach (Activity currentActivity in activities.Where(a => a.Type == ActivityTypes.Message))
{
ConvertTextToSpeech(currentActivity);
}
return next();
}
private void ConvertTextToSpeech(Activity message)
{
Activity initialMessage = message;
try
{
if (message.Type == ActivityTypes.Message)
{
if (string.IsNullOrEmpty(message.Speak))
{
if (string.IsNullOrEmpty(message.Text))
{
if (message.Attachments[0].Content is HeroCard attachment)
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(attachment.Text);
}
}
else
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(message.Text);
}
message.Speak = message.Speak.Trim();
if (ignoreList.Where(i => message.Speak.ToLower().StartsWith(i.ToLower())).Count() != 0)
{
message.Speak = null;
ignoredSpeak = true;
}
}
else if (string.IsNullOrWhiteSpace(message.Speak))
{
message.Speak = null;
}
if ((!string.IsNullOrEmpty(message.Speak) && (message.Speak.EndsWith("?") || message.Speak.StartsWith("Is this correct?")))
|| (!string.IsNullOrEmpty(message.Text) && message.Text.EndsWith("?")))
{
message.InputHint = InputHints.ExpectingInput;
}
// IOs won't work with expecting input
if (message.Recipient.Name.EndsWith(":ios"))
{
message.InputHint = InputHints.AcceptingInput;
}
}
}
catch (Exception)
{
message = initialMessage;
}
}
}
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
add a comment |
If you want the mic to turn on after the bot speaks to you, you need to set the inputHint on the activity to ExpectingInput. If you always want that and you don't send multiple messages at a time, then you can set it with an ActivityMapper (v3) or a Middleware (v4). Just a note that with v3 the ExpectingInput breaks on ios. I'm not sure about v4 though. Below is an example middleware (v4) I have that sets the inputHint to ExpectingInput if the text or attachment ends with a "?".
public class TextToSpeechMiddleware : IMiddleware
{
public Task OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken = default(CancellationToken))
{
turnContext.OnSendActivities(OnSendActivities);
turnContext.OnUpdateActivity(OnUpdateActivity);
return next(cancellationToken);
}
private Task<ResourceResponse> OnUpdateActivity(ITurnContext turnContext, Activity activity, Func<Task<ResourceResponse>> next)
{
ConvertTextToSpeech(activity);
return next();
}
private Task<ResourceResponse> OnSendActivities(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse>> next)
{
foreach (Activity currentActivity in activities.Where(a => a.Type == ActivityTypes.Message))
{
ConvertTextToSpeech(currentActivity);
}
return next();
}
private void ConvertTextToSpeech(Activity message)
{
Activity initialMessage = message;
try
{
if (message.Type == ActivityTypes.Message)
{
if (string.IsNullOrEmpty(message.Speak))
{
if (string.IsNullOrEmpty(message.Text))
{
if (message.Attachments[0].Content is HeroCard attachment)
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(attachment.Text);
}
}
else
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(message.Text);
}
message.Speak = message.Speak.Trim();
if (ignoreList.Where(i => message.Speak.ToLower().StartsWith(i.ToLower())).Count() != 0)
{
message.Speak = null;
ignoredSpeak = true;
}
}
else if (string.IsNullOrWhiteSpace(message.Speak))
{
message.Speak = null;
}
if ((!string.IsNullOrEmpty(message.Speak) && (message.Speak.EndsWith("?") || message.Speak.StartsWith("Is this correct?")))
|| (!string.IsNullOrEmpty(message.Text) && message.Text.EndsWith("?")))
{
message.InputHint = InputHints.ExpectingInput;
}
// IOs won't work with expecting input
if (message.Recipient.Name.EndsWith(":ios"))
{
message.InputHint = InputHints.AcceptingInput;
}
}
}
catch (Exception)
{
message = initialMessage;
}
}
}
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
add a comment |
If you want the mic to turn on after the bot speaks to you, you need to set the inputHint on the activity to ExpectingInput. If you always want that and you don't send multiple messages at a time, then you can set it with an ActivityMapper (v3) or a Middleware (v4). Just a note that with v3 the ExpectingInput breaks on ios. I'm not sure about v4 though. Below is an example middleware (v4) I have that sets the inputHint to ExpectingInput if the text or attachment ends with a "?".
public class TextToSpeechMiddleware : IMiddleware
{
public Task OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken = default(CancellationToken))
{
turnContext.OnSendActivities(OnSendActivities);
turnContext.OnUpdateActivity(OnUpdateActivity);
return next(cancellationToken);
}
private Task<ResourceResponse> OnUpdateActivity(ITurnContext turnContext, Activity activity, Func<Task<ResourceResponse>> next)
{
ConvertTextToSpeech(activity);
return next();
}
private Task<ResourceResponse> OnSendActivities(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse>> next)
{
foreach (Activity currentActivity in activities.Where(a => a.Type == ActivityTypes.Message))
{
ConvertTextToSpeech(currentActivity);
}
return next();
}
private void ConvertTextToSpeech(Activity message)
{
Activity initialMessage = message;
try
{
if (message.Type == ActivityTypes.Message)
{
if (string.IsNullOrEmpty(message.Speak))
{
if (string.IsNullOrEmpty(message.Text))
{
if (message.Attachments[0].Content is HeroCard attachment)
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(attachment.Text);
}
}
else
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(message.Text);
}
message.Speak = message.Speak.Trim();
if (ignoreList.Where(i => message.Speak.ToLower().StartsWith(i.ToLower())).Count() != 0)
{
message.Speak = null;
ignoredSpeak = true;
}
}
else if (string.IsNullOrWhiteSpace(message.Speak))
{
message.Speak = null;
}
if ((!string.IsNullOrEmpty(message.Speak) && (message.Speak.EndsWith("?") || message.Speak.StartsWith("Is this correct?")))
|| (!string.IsNullOrEmpty(message.Text) && message.Text.EndsWith("?")))
{
message.InputHint = InputHints.ExpectingInput;
}
// IOs won't work with expecting input
if (message.Recipient.Name.EndsWith(":ios"))
{
message.InputHint = InputHints.AcceptingInput;
}
}
}
catch (Exception)
{
message = initialMessage;
}
}
}
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
If you want the mic to turn on after the bot speaks to you, you need to set the inputHint on the activity to ExpectingInput. If you always want that and you don't send multiple messages at a time, then you can set it with an ActivityMapper (v3) or a Middleware (v4). Just a note that with v3 the ExpectingInput breaks on ios. I'm not sure about v4 though. Below is an example middleware (v4) I have that sets the inputHint to ExpectingInput if the text or attachment ends with a "?".
public class TextToSpeechMiddleware : IMiddleware
{
public Task OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken = default(CancellationToken))
{
turnContext.OnSendActivities(OnSendActivities);
turnContext.OnUpdateActivity(OnUpdateActivity);
return next(cancellationToken);
}
private Task<ResourceResponse> OnUpdateActivity(ITurnContext turnContext, Activity activity, Func<Task<ResourceResponse>> next)
{
ConvertTextToSpeech(activity);
return next();
}
private Task<ResourceResponse> OnSendActivities(ITurnContext turnContext, List<Activity> activities, Func<Task<ResourceResponse>> next)
{
foreach (Activity currentActivity in activities.Where(a => a.Type == ActivityTypes.Message))
{
ConvertTextToSpeech(currentActivity);
}
return next();
}
private void ConvertTextToSpeech(Activity message)
{
Activity initialMessage = message;
try
{
if (message.Type == ActivityTypes.Message)
{
if (string.IsNullOrEmpty(message.Speak))
{
if (string.IsNullOrEmpty(message.Text))
{
if (message.Attachments[0].Content is HeroCard attachment)
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(attachment.Text);
}
}
else
{
message.Speak = TextToSpeechHelper.ConvertTextToSpeechText(message.Text);
}
message.Speak = message.Speak.Trim();
if (ignoreList.Where(i => message.Speak.ToLower().StartsWith(i.ToLower())).Count() != 0)
{
message.Speak = null;
ignoredSpeak = true;
}
}
else if (string.IsNullOrWhiteSpace(message.Speak))
{
message.Speak = null;
}
if ((!string.IsNullOrEmpty(message.Speak) && (message.Speak.EndsWith("?") || message.Speak.StartsWith("Is this correct?")))
|| (!string.IsNullOrEmpty(message.Text) && message.Text.EndsWith("?")))
{
message.InputHint = InputHints.ExpectingInput;
}
// IOs won't work with expecting input
if (message.Recipient.Name.EndsWith(":ios"))
{
message.InputHint = InputHints.AcceptingInput;
}
}
}
catch (Exception)
{
message = initialMessage;
}
}
}
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
answered Nov 20 '18 at 19:56
kjr1995kjr1995
305138
305138
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388242%2fspeech-to-text-in-microsoft-botframework-using-c-sharp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Are you using v3 or v4 of the bot framework?
– kjr1995
Nov 20 '18 at 14:35