Vjylku

Cauchy Sequence Characterized only By Directly Neighbouring Sequence Members

I'm having difficulty getting my players to do stuff in a sandbox campaign

What are the performance impacts of 'functional' Rust?

Why is "Captain Marvel" translated as male in Portugal?

Area of a 2D convex hull

Why is there no army of Iron-Mans in the MCU?

What is the electric potential inside a point charge?

How is simplicity better than precision and clarity in prose?

How do you clear the ApexPages.getMessages() collection in a test?

Jazz greats knew nothing of modes. Why are they used to improvise on standards?

What do you call the holes in a flute?

Autumning in love

Passing functions in C++

Strange behaviour of Check

How to politely respond to generic emails requesting a PhD/job in my lab? Without wasting too much time

How should I respond to a player wanting to catch a sword between their hands?

Complexity of many constant time steps with occasional logarithmic steps

What LEGO pieces have "real-world" functionality?

How to rotate it perfectly?

If I can make up priors, why can't I make up posteriors?

Windows 10: How to Lock (not sleep) laptop on lid close?

Is there a service that would inform me whenever a new direct route is scheduled from a given airport?

Cold is to Refrigerator as warm is to?

Can a non-EU citizen traveling with me come with me through the EU passport line?

What loss function to use when labels are probabilities?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this question

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 5 hours ago

Thomas JohnsonThomas Johnson

1083

1 Answer
1

active

oldest

votes

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "658"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

Thomas Johnson is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fai.stackexchange.com%2fquestions%2f11816%2fwhat-loss-function-to-use-when-labels-are-probabilities%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119