phonemes generated on-the-fly by espeak-ng (GPL license)

[gyroing]

Some languages, like Persian, are highly dependent on the wordrule of eSpeak for phoneme generation. Unfortunately, the output quality is not good.

I wish you had made a version where phonemes are still generated on-the-fly by eSpeak.

[nihui]

Yes, espeak-ng performs better, but espeak-ng is licensed under the GPL, which would force this project to also adopt the GPL, which is detrimental to free commercial use.

Here is a GPL-licensed implementation of phonemize using espeak-ng.

If you use it, you must open source your project under the GPL.

typedef char32_t Phoneme;
typedef int64_t PhonemeId;
typedef int64_t SpeakerId;
typedef std::map<Phoneme, std::vector<PhonemeId>> PhonemeIdMap;

// Get the first UTF-8 codepoint of a string
std::optional<Phoneme> get_codepoint(std::string s)
{
    auto view = una::views::utf8(s);
    auto it = view.begin();

    if (it != view.end()) {
        return *it;
    }

    return std::nullopt;
}

void phonemize(const char *text, ncnn::Mat& sequence)
{
    fprintf(stderr, "text = %s\n", text);

#define CLAUSE_INTONATION_FULL_STOP 0x00000000
#define CLAUSE_INTONATION_COMMA     0x00001000
#define CLAUSE_INTONATION_QUESTION  0x00002000
#define CLAUSE_INTONATION_EXCLAMATION 0x00003000

#define CLAUSE_TYPE_CLAUSE   0x00040000
#define CLAUSE_TYPE_SENTENCE 0x00080000

#define CLAUSE_PERIOD       (40 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COMMA        (20 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_QUESTION     (40 | CLAUSE_INTONATION_QUESTION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_EXCLAMATION  (45 | CLAUSE_INTONATION_EXCLAMATION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COLON        (30 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_SEMICOLON    (30 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)

    const char* data_dir = "espeak-ng/build/install/share/espeak-ng-data";
    int ret = espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, data_dir, 0);
    fprintf(stderr, "espeak_Initialize = %d\n", ret);

    const char* voice = "en-us";
    int ret1 = espeak_SetVoiceByName(voice);
    fprintf(stderr, "espeak_SetVoiceByName = %d\n", ret1);

    // phonemize
    std::vector<std::string> sentence_phonemes{""};
    std::size_t current_idx = 0;
    const void *text_ptr = text;
    while (text_ptr != nullptr)
    {
        int terminator = 0;
        std::string terminator_str = "";

        const char *phonemes = espeak_TextToPhonemesWithTerminator(&text_ptr, espeakCHARS_AUTO, espeakPHONEMES_IPA, &terminator);

        fprintf(stderr, "phonemes = %s\n", phonemes);

        if (phonemes)
        {
            sentence_phonemes[current_idx] += phonemes;
        }

        // Categorize terminator
        terminator &= 0x000FFFFF;

        if (terminator == CLAUSE_PERIOD) {
            terminator_str = ".";
        } else if (terminator == CLAUSE_QUESTION) {
            terminator_str = "?";
        } else if (terminator == CLAUSE_EXCLAMATION) {
            terminator_str = "!";
        } else if (terminator == CLAUSE_COMMA) {
            terminator_str = ", ";
        } else if (terminator == CLAUSE_COLON) {
            terminator_str = ": ";
        } else if (terminator == CLAUSE_SEMICOLON) {
            terminator_str = "; ";
        }

        sentence_phonemes[current_idx] += terminator_str;

        if ((terminator & CLAUSE_TYPE_SENTENCE) == CLAUSE_TYPE_SENTENCE) {
            sentence_phonemes.push_back("");
            current_idx = sentence_phonemes.size() - 1;
        }
    }

    // phonemes to ids
    PhonemeIdMap phoneme_id_map;
    // synthesize state
    std::vector<std::vector<PhonemeId>> phoneme_id_queue;

    const PhonemeId ID_PAD = 0; // interleaved
    const PhonemeId ID_BOS = 1; // beginning of sentence
    const PhonemeId ID_EOS = 2; // end of sentence

    std::ifstream config_stream("zh-config.json");
    auto config = nlohmann::json::parse(config_stream);

    // phoneme to [id] map
    // Maps phonemes to one or more phoneme ids (required).
    if (config.contains("phoneme_id_map")) {
        auto &phoneme_id_map_value = config["phoneme_id_map"];
        for (auto &from_phoneme_item : phoneme_id_map_value.items()) {
            std::string from_phoneme = from_phoneme_item.key();
            auto from_codepoint = get_codepoint(from_phoneme);
            if (!from_codepoint) {
                // No codepoint
                continue;
            }

            for (auto &to_id_value : from_phoneme_item.value()) {
                PhonemeId to_id = to_id_value.get<PhonemeId>();
                phoneme_id_map[*from_codepoint].push_back(to_id);
            }
        }
    }

    std::vector<PhonemeId> sentence_ids;
    for (auto &phonemes_str : sentence_phonemes)
    {
        if (phonemes_str.empty()) {
            continue;
        }

        sentence_ids.push_back(ID_BOS);
        sentence_ids.push_back(ID_PAD);

        auto phonemes_norm = una::norm::to_nfd_utf8(phonemes_str);
        auto phonemes_range = una::ranges::utf8_view{phonemes_norm};
        auto phonemes_iter = phonemes_range.begin();
        auto phonemes_end = phonemes_range.end();

        // Filter out (lang) switch (flags).
        // These surround words from languages other than the current voice.
        bool in_lang_flag = false;
        while (phonemes_iter != phonemes_end)
        {
            if (in_lang_flag)
            {
                if (*phonemes_iter == U')')
                {
                    // End of (lang) switch
                    in_lang_flag = false;
                }
            }
            else if (*phonemes_iter == U'(')
            {
                // Start of (lang) switch
                in_lang_flag = true;
            }
            else
            {
                // Look up ids
                auto ids_for_phoneme = phoneme_id_map.find(*phonemes_iter);
                if (ids_for_phoneme != phoneme_id_map.end())
                {
                    for (auto id : ids_for_phoneme->second)
                    {
                        sentence_ids.push_back(id);
                        sentence_ids.push_back(ID_PAD);
                    }
                }
            }

            phonemes_iter++;
        }

        sentence_ids.push_back(ID_EOS);

        phoneme_id_queue.push_back(sentence_ids);
        sentence_ids.clear();
    }


    fprintf(stderr, "phoneme_id_queue.size() = %d\n", (int)phoneme_id_queue.size());

    std::vector<int> sequence_combined;

    for (int i = 0; i < phoneme_id_queue.size(); i++)
    {
        for (int j = 0; j < phoneme_id_queue[i].size(); j++)
        {
            // fprintf(stderr, "%d\n", phoneme_id_queue[i][j]);
            // ps[j] = phoneme_id_queue[i][j];
            sequence_combined.push_back(phoneme_id_queue[i][j]);
        }
    }

    int sequence_length = sequence_combined.size();

    sequence.create(sequence_length);
    int* ps = sequence;
    for (int i = 0; i < sequence_length; i++)
    {
        ps[i] = sequence_combined[i];
    }

    espeak_Terminate();
}

[gyroing]

Yes, espeak-ng performs better, but espeak-ng is licensed under the GPL, which would force this project to also adopt the GPL, which is detrimental to free commercial use.

Here is a GPL-licensed implementation of phonemize using espeak-ng.

If you use it, you must open source your project under the GPL.

typedef char32_t Phoneme;
typedef int64_t PhonemeId;
typedef int64_t SpeakerId;
typedef std::map<Phoneme, std::vector> PhonemeIdMap;

// Get the first UTF-8 codepoint of a string
std::optional get_codepoint(std::string s)
{
auto view = una::views::utf8(s);
auto it = view.begin();

if (it != view.end()) {
    return *it;
}

return std::nullopt;

}

void phonemize(const char *text, ncnn::Mat& sequence)
{
fprintf(stderr, "text = %s\n", text);

#define CLAUSE_INTONATION_FULL_STOP 0x00000000
#define CLAUSE_INTONATION_COMMA 0x00001000
#define CLAUSE_INTONATION_QUESTION 0x00002000
#define CLAUSE_INTONATION_EXCLAMATION 0x00003000

#define CLAUSE_TYPE_CLAUSE 0x00040000
#define CLAUSE_TYPE_SENTENCE 0x00080000

#define CLAUSE_PERIOD (40 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COMMA (20 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_QUESTION (40 | CLAUSE_INTONATION_QUESTION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_EXCLAMATION (45 | CLAUSE_INTONATION_EXCLAMATION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COLON (30 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_SEMICOLON (30 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)

const char* data_dir = "espeak-ng/build/install/share/espeak-ng-data";
int ret = espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, data_dir, 0);
fprintf(stderr, "espeak_Initialize = %d\n", ret);

const char* voice = "en-us";
int ret1 = espeak_SetVoiceByName(voice);
fprintf(stderr, "espeak_SetVoiceByName = %d\n", ret1);

// phonemize
std::vector<std::string> sentence_phonemes{""};
std::size_t current_idx = 0;
const void *text_ptr = text;
while (text_ptr != nullptr)
{
    int terminator = 0;
    std::string terminator_str = "";

    const char *phonemes = espeak_TextToPhonemesWithTerminator(&text_ptr, espeakCHARS_AUTO, espeakPHONEMES_IPA, &terminator);

    fprintf(stderr, "phonemes = %s\n", phonemes);

    if (phonemes)
    {
        sentence_phonemes[current_idx] += phonemes;
    }

    // Categorize terminator
    terminator &= 0x000FFFFF;

    if (terminator == CLAUSE_PERIOD) {
        terminator_str = ".";
    } else if (terminator == CLAUSE_QUESTION) {
        terminator_str = "?";
    } else if (terminator == CLAUSE_EXCLAMATION) {
        terminator_str = "!";
    } else if (terminator == CLAUSE_COMMA) {
        terminator_str = ", ";
    } else if (terminator == CLAUSE_COLON) {
        terminator_str = ": ";
    } else if (terminator == CLAUSE_SEMICOLON) {
        terminator_str = "; ";
    }

    sentence_phonemes[current_idx] += terminator_str;

    if ((terminator & CLAUSE_TYPE_SENTENCE) == CLAUSE_TYPE_SENTENCE) {
        sentence_phonemes.push_back("");
        current_idx = sentence_phonemes.size() - 1;
    }
}

// phonemes to ids
PhonemeIdMap phoneme_id_map;
// synthesize state
std::vector<std::vector<PhonemeId>> phoneme_id_queue;

const PhonemeId ID_PAD = 0; // interleaved
const PhonemeId ID_BOS = 1; // beginning of sentence
const PhonemeId ID_EOS = 2; // end of sentence

std::ifstream config_stream("zh-config.json");
auto config = nlohmann::json::parse(config_stream);

// phoneme to [id] map
// Maps phonemes to one or more phoneme ids (required).
if (config.contains("phoneme_id_map")) {
    auto &phoneme_id_map_value = config["phoneme_id_map"];
    for (auto &from_phoneme_item : phoneme_id_map_value.items()) {
        std::string from_phoneme = from_phoneme_item.key();
        auto from_codepoint = get_codepoint(from_phoneme);
        if (!from_codepoint) {
            // No codepoint
            continue;
        }

        for (auto &to_id_value : from_phoneme_item.value()) {
            PhonemeId to_id = to_id_value.get<PhonemeId>();
            phoneme_id_map[*from_codepoint].push_back(to_id);
        }
    }
}

std::vector<PhonemeId> sentence_ids;
for (auto &phonemes_str : sentence_phonemes)
{
    if (phonemes_str.empty()) {
        continue;
    }

    sentence_ids.push_back(ID_BOS);
    sentence_ids.push_back(ID_PAD);

    auto phonemes_norm = una::norm::to_nfd_utf8(phonemes_str);
    auto phonemes_range = una::ranges::utf8_view{phonemes_norm};
    auto phonemes_iter = phonemes_range.begin();
    auto phonemes_end = phonemes_range.end();

    // Filter out (lang) switch (flags).
    // These surround words from languages other than the current voice.
    bool in_lang_flag = false;
    while (phonemes_iter != phonemes_end)
    {
        if (in_lang_flag)
        {
            if (*phonemes_iter == U')')
            {
                // End of (lang) switch
                in_lang_flag = false;
            }
        }
        else if (*phonemes_iter == U'(')
        {
            // Start of (lang) switch
            in_lang_flag = true;
        }
        else
        {
            // Look up ids
            auto ids_for_phoneme = phoneme_id_map.find(*phonemes_iter);
            if (ids_for_phoneme != phoneme_id_map.end())
            {
                for (auto id : ids_for_phoneme->second)
                {
                    sentence_ids.push_back(id);
                    sentence_ids.push_back(ID_PAD);
                }
            }
        }

        phonemes_iter++;
    }

    sentence_ids.push_back(ID_EOS);

    phoneme_id_queue.push_back(sentence_ids);
    sentence_ids.clear();
}


fprintf(stderr, "phoneme_id_queue.size() = %d\n", (int)phoneme_id_queue.size());

std::vector<int> sequence_combined;

for (int i = 0; i < phoneme_id_queue.size(); i++)
{
    for (int j = 0; j < phoneme_id_queue[i].size(); j++)
    {
        // fprintf(stderr, "%d\n", phoneme_id_queue[i][j]);
        // ps[j] = phoneme_id_queue[i][j];
        sequence_combined.push_back(phoneme_id_queue[i][j]);
    }
}

int sequence_length = sequence_combined.size();

sequence.create(sequence_length);
int* ps = sequence;
for (int i = 0; i < sequence_length; i++)
{
    ps[i] = sequence_combined[i];
}

espeak_Terminate();

}

I just wanted to let you know that I noticed your project includes code from OHF-Voice/piper1-gpl. That project is released under the GPL license, which means your code that uses it also falls under GPL.

"I believe some of this code is also used in your JNI part."

piper.cpp piper.h and so...

[nihui]

Yes, espeak-ng performs better, but espeak-ng is licensed under the GPL, which would force this project to also adopt the GPL, which is detrimental to free commercial use.
Here is a GPL-licensed implementation of phonemize using espeak-ng.
If you use it, you must open source your project under the GPL.
typedef char32_t Phoneme;
typedef int64_t PhonemeId;
typedef int64_t SpeakerId;
typedef std::map<Phoneme, std::vector> PhonemeIdMap;
// Get the first UTF-8 codepoint of a string
std::optional get_codepoint(std::string s)
{
auto view = una::views::utf8(s);
auto it = view.begin();

if (it != view.end()) {
    return *it;
}

return std::nullopt;

}
void phonemize(const char *text, ncnn::Mat& sequence)
{
fprintf(stderr, "text = %s\n", text);
#define CLAUSE_INTONATION_FULL_STOP 0x00000000
#define CLAUSE_INTONATION_COMMA 0x00001000
#define CLAUSE_INTONATION_QUESTION 0x00002000
#define CLAUSE_INTONATION_EXCLAMATION 0x00003000
#define CLAUSE_TYPE_CLAUSE 0x00040000
#define CLAUSE_TYPE_SENTENCE 0x00080000
#define CLAUSE_PERIOD (40 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COMMA (20 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_QUESTION (40 | CLAUSE_INTONATION_QUESTION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_EXCLAMATION (45 | CLAUSE_INTONATION_EXCLAMATION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COLON (30 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_SEMICOLON (30 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)

const char* data_dir = "espeak-ng/build/install/share/espeak-ng-data";
int ret = espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, data_dir, 0);
fprintf(stderr, "espeak_Initialize = %d\n", ret);

const char* voice = "en-us";
int ret1 = espeak_SetVoiceByName(voice);
fprintf(stderr, "espeak_SetVoiceByName = %d\n", ret1);

// phonemize
std::vector<std::string> sentence_phonemes{""};
std::size_t current_idx = 0;
const void *text_ptr = text;
while (text_ptr != nullptr)
{
    int terminator = 0;
    std::string terminator_str = "";

    const char *phonemes = espeak_TextToPhonemesWithTerminator(&text_ptr, espeakCHARS_AUTO, espeakPHONEMES_IPA, &terminator);

    fprintf(stderr, "phonemes = %s\n", phonemes);

    if (phonemes)
    {
        sentence_phonemes[current_idx] += phonemes;
    }

    // Categorize terminator
    terminator &= 0x000FFFFF;

    if (terminator == CLAUSE_PERIOD) {
        terminator_str = ".";
    } else if (terminator == CLAUSE_QUESTION) {
        terminator_str = "?";
    } else if (terminator == CLAUSE_EXCLAMATION) {
        terminator_str = "!";
    } else if (terminator == CLAUSE_COMMA) {
        terminator_str = ", ";
    } else if (terminator == CLAUSE_COLON) {
        terminator_str = ": ";
    } else if (terminator == CLAUSE_SEMICOLON) {
        terminator_str = "; ";
    }

    sentence_phonemes[current_idx] += terminator_str;

    if ((terminator & CLAUSE_TYPE_SENTENCE) == CLAUSE_TYPE_SENTENCE) {
        sentence_phonemes.push_back("");
        current_idx = sentence_phonemes.size() - 1;
    }
}

// phonemes to ids
PhonemeIdMap phoneme_id_map;
// synthesize state
std::vector<std::vector<PhonemeId>> phoneme_id_queue;

const PhonemeId ID_PAD = 0; // interleaved
const PhonemeId ID_BOS = 1; // beginning of sentence
const PhonemeId ID_EOS = 2; // end of sentence

std::ifstream config_stream("zh-config.json");
auto config = nlohmann::json::parse(config_stream);

// phoneme to [id] map
// Maps phonemes to one or more phoneme ids (required).
if (config.contains("phoneme_id_map")) {
    auto &phoneme_id_map_value = config["phoneme_id_map"];
    for (auto &from_phoneme_item : phoneme_id_map_value.items()) {
        std::string from_phoneme = from_phoneme_item.key();
        auto from_codepoint = get_codepoint(from_phoneme);
        if (!from_codepoint) {
            // No codepoint
            continue;
        }

        for (auto &to_id_value : from_phoneme_item.value()) {
            PhonemeId to_id = to_id_value.get<PhonemeId>();
            phoneme_id_map[*from_codepoint].push_back(to_id);
        }
    }
}

std::vector<PhonemeId> sentence_ids;
for (auto &phonemes_str : sentence_phonemes)
{
    if (phonemes_str.empty()) {
        continue;
    }

    sentence_ids.push_back(ID_BOS);
    sentence_ids.push_back(ID_PAD);

    auto phonemes_norm = una::norm::to_nfd_utf8(phonemes_str);
    auto phonemes_range = una::ranges::utf8_view{phonemes_norm};
    auto phonemes_iter = phonemes_range.begin();
    auto phonemes_end = phonemes_range.end();

    // Filter out (lang) switch (flags).
    // These surround words from languages other than the current voice.
    bool in_lang_flag = false;
    while (phonemes_iter != phonemes_end)
    {
        if (in_lang_flag)
        {
            if (*phonemes_iter == U')')
            {
                // End of (lang) switch
                in_lang_flag = false;
            }
        }
        else if (*phonemes_iter == U'(')
        {
            // Start of (lang) switch
            in_lang_flag = true;
        }
        else
        {
            // Look up ids
            auto ids_for_phoneme = phoneme_id_map.find(*phonemes_iter);
            if (ids_for_phoneme != phoneme_id_map.end())
            {
                for (auto id : ids_for_phoneme->second)
                {
                    sentence_ids.push_back(id);
                    sentence_ids.push_back(ID_PAD);
                }
            }
        }

        phonemes_iter++;
    }

    sentence_ids.push_back(ID_EOS);

    phoneme_id_queue.push_back(sentence_ids);
    sentence_ids.clear();
}


fprintf(stderr, "phoneme_id_queue.size() = %d\n", (int)phoneme_id_queue.size());

std::vector<int> sequence_combined;

for (int i = 0; i < phoneme_id_queue.size(); i++)
{
    for (int j = 0; j < phoneme_id_queue[i].size(); j++)
    {
        // fprintf(stderr, "%d\n", phoneme_id_queue[i][j]);
        // ps[j] = phoneme_id_queue[i][j];
        sequence_combined.push_back(phoneme_id_queue[i][j]);
    }
}

int sequence_length = sequence_combined.size();

sequence.create(sequence_length);
int* ps = sequence;
for (int i = 0; i < sequence_length; i++)
{
    ps[i] = sequence_combined[i];
}

espeak_Terminate();

}

I just wanted to let you know that I noticed your project includes code from OHF-Voice/piper1-gpl. That project is released under the GPL license, which means your code that uses it also falls under GPL.

"I believe some of this code is also used in your JNI part."

piper.cpp piper.h and so...

This project reimplements the piper algorithm using the ncnn framework and does not contain code from the piper-GPL project.

Please list the specific code files you believe are suspicious and the reasons for your determination.

[gyroing]

Yes, espeak-ng performs better, but espeak-ng is licensed under the GPL, which would force this project to also adopt the GPL, which is detrimental to free commercial use.
Here is a GPL-licensed implementation of phonemize using espeak-ng.
If you use it, you must open source your project under the GPL.
typedef char32_t Phoneme;
typedef int64_t PhonemeId;
typedef int64_t SpeakerId;
typedef std::map<Phoneme, std::vector> PhonemeIdMap;
// Get the first UTF-8 codepoint of a string
std::optional get_codepoint(std::string s)
{
auto view = una::views::utf8(s);
auto it = view.begin();

if (it != view.end()) {
    return *it;
}

return std::nullopt;

}
void phonemize(const char *text, ncnn::Mat& sequence)
{
fprintf(stderr, "text = %s\n", text);
#define CLAUSE_INTONATION_FULL_STOP 0x00000000
#define CLAUSE_INTONATION_COMMA 0x00001000
#define CLAUSE_INTONATION_QUESTION 0x00002000
#define CLAUSE_INTONATION_EXCLAMATION 0x00003000
#define CLAUSE_TYPE_CLAUSE 0x00040000
#define CLAUSE_TYPE_SENTENCE 0x00080000
#define CLAUSE_PERIOD (40 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COMMA (20 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_QUESTION (40 | CLAUSE_INTONATION_QUESTION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_EXCLAMATION (45 | CLAUSE_INTONATION_EXCLAMATION | CLAUSE_TYPE_SENTENCE)
#define CLAUSE_COLON (30 | CLAUSE_INTONATION_FULL_STOP | CLAUSE_TYPE_CLAUSE)
#define CLAUSE_SEMICOLON (30 | CLAUSE_INTONATION_COMMA | CLAUSE_TYPE_CLAUSE)

const char* data_dir = "espeak-ng/build/install/share/espeak-ng-data";
int ret = espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, data_dir, 0);
fprintf(stderr, "espeak_Initialize = %d\n", ret);

const char* voice = "en-us";
int ret1 = espeak_SetVoiceByName(voice);
fprintf(stderr, "espeak_SetVoiceByName = %d\n", ret1);

// phonemize
std::vector<std::string> sentence_phonemes{""};
std::size_t current_idx = 0;
const void *text_ptr = text;
while (text_ptr != nullptr)
{
    int terminator = 0;
    std::string terminator_str = "";

    const char *phonemes = espeak_TextToPhonemesWithTerminator(&text_ptr, espeakCHARS_AUTO, espeakPHONEMES_IPA, &terminator);

    fprintf(stderr, "phonemes = %s\n", phonemes);

    if (phonemes)
    {
        sentence_phonemes[current_idx] += phonemes;
    }

    // Categorize terminator
    terminator &= 0x000FFFFF;

    if (terminator == CLAUSE_PERIOD) {
        terminator_str = ".";
    } else if (terminator == CLAUSE_QUESTION) {
        terminator_str = "?";
    } else if (terminator == CLAUSE_EXCLAMATION) {
        terminator_str = "!";
    } else if (terminator == CLAUSE_COMMA) {
        terminator_str = ", ";
    } else if (terminator == CLAUSE_COLON) {
        terminator_str = ": ";
    } else if (terminator == CLAUSE_SEMICOLON) {
        terminator_str = "; ";
    }

    sentence_phonemes[current_idx] += terminator_str;

    if ((terminator & CLAUSE_TYPE_SENTENCE) == CLAUSE_TYPE_SENTENCE) {
        sentence_phonemes.push_back("");
        current_idx = sentence_phonemes.size() - 1;
    }
}

// phonemes to ids
PhonemeIdMap phoneme_id_map;
// synthesize state
std::vector<std::vector<PhonemeId>> phoneme_id_queue;

const PhonemeId ID_PAD = 0; // interleaved
const PhonemeId ID_BOS = 1; // beginning of sentence
const PhonemeId ID_EOS = 2; // end of sentence

std::ifstream config_stream("zh-config.json");
auto config = nlohmann::json::parse(config_stream);

// phoneme to [id] map
// Maps phonemes to one or more phoneme ids (required).
if (config.contains("phoneme_id_map")) {
    auto &phoneme_id_map_value = config["phoneme_id_map"];
    for (auto &from_phoneme_item : phoneme_id_map_value.items()) {
        std::string from_phoneme = from_phoneme_item.key();
        auto from_codepoint = get_codepoint(from_phoneme);
        if (!from_codepoint) {
            // No codepoint
            continue;
        }

        for (auto &to_id_value : from_phoneme_item.value()) {
            PhonemeId to_id = to_id_value.get<PhonemeId>();
            phoneme_id_map[*from_codepoint].push_back(to_id);
        }
    }
}

std::vector<PhonemeId> sentence_ids;
for (auto &phonemes_str : sentence_phonemes)
{
    if (phonemes_str.empty()) {
        continue;
    }

    sentence_ids.push_back(ID_BOS);
    sentence_ids.push_back(ID_PAD);

    auto phonemes_norm = una::norm::to_nfd_utf8(phonemes_str);
    auto phonemes_range = una::ranges::utf8_view{phonemes_norm};
    auto phonemes_iter = phonemes_range.begin();
    auto phonemes_end = phonemes_range.end();

    // Filter out (lang) switch (flags).
    // These surround words from languages other than the current voice.
    bool in_lang_flag = false;
    while (phonemes_iter != phonemes_end)
    {
        if (in_lang_flag)
        {
            if (*phonemes_iter == U')')
            {
                // End of (lang) switch
                in_lang_flag = false;
            }
        }
        else if (*phonemes_iter == U'(')
        {
            // Start of (lang) switch
            in_lang_flag = true;
        }
        else
        {
            // Look up ids
            auto ids_for_phoneme = phoneme_id_map.find(*phonemes_iter);
            if (ids_for_phoneme != phoneme_id_map.end())
            {
                for (auto id : ids_for_phoneme->second)
                {
                    sentence_ids.push_back(id);
                    sentence_ids.push_back(ID_PAD);
                }
            }
        }

        phonemes_iter++;
    }

    sentence_ids.push_back(ID_EOS);

    phoneme_id_queue.push_back(sentence_ids);
    sentence_ids.clear();
}


fprintf(stderr, "phoneme_id_queue.size() = %d\n", (int)phoneme_id_queue.size());

std::vector<int> sequence_combined;

for (int i = 0; i < phoneme_id_queue.size(); i++)
{
    for (int j = 0; j < phoneme_id_queue[i].size(); j++)
    {
        // fprintf(stderr, "%d\n", phoneme_id_queue[i][j]);
        // ps[j] = phoneme_id_queue[i][j];
        sequence_combined.push_back(phoneme_id_queue[i][j]);
    }
}

int sequence_length = sequence_combined.size();

sequence.create(sequence_length);
int* ps = sequence;
for (int i = 0; i < sequence_length; i++)
{
    ps[i] = sequence_combined[i];
}

espeak_Terminate();

}

I just wanted to let you know that I noticed your project includes code from OHF-Voice/piper1-gpl. That project is released under the GPL license, which means your code that uses it also falls under GPL.
"I believe some of this code is also used in your JNI part."
piper.cpp piper.h and so...

This project reimplements the piper algorithm using the ncnn framework and does not contain code from the piper-GPL project.

Please list the specific code files you believe are suspicious and the reasons for your determination.

Yes, you are right — I looked more carefully and I see that the codes are also rewritten. 🙏

[makiapplpretzele]

Hi, nice project! 😁 Could a sort-of straight forward instruction in implementing this eSpeak-NG way of phonemizing be made for people who are not yet that proficient with using eSpeak-NG, CMAKE or C++ overall? I'm trying to achieve a customized version of this nice project to be able to run on Android Debug shell, without a JNI, (as in a binary) and the eSpeak-NG phonemizer would be great and essential for this. So far, I understand some to configure the espeak-ng-data directory, and others things, but not yet have understood how to integrate eSpeak-NG to the implementation. Research should be done of course, to try to find information and all, but was wondering if you already got it figured. Thank you. 😊😁