{"id":8668,"date":"2019-11-30T17:23:25","date_gmt":"2019-11-30T08:23:25","guid":{"rendered":"http:\/\/www.gisdeveloper.co.kr\/?p=8668"},"modified":"2020-05-28T09:52:28","modified_gmt":"2020-05-28T00:52:28","slug":"torch-nn-crossentropylos","status":"publish","type":"post","link":"http:\/\/www.gisdeveloper.co.kr\/?p=8668","title":{"rendered":"\ub2e4\uc911\ubd84\ub958\ub97c \uc704\ud55c \ub300\ud45c\uc801\uc778 \uc190\uc2e4\ud568\uc218, torch.nn.CrossEntropyLoss"},"content":{"rendered":"<p>\ub525\ub7ec\ub2dd\uc758 \ub9ce\uc740 \uc774\ub860 \uc911 \uac00\uc7a5 \uc911\uc694\ud55c \ubd80\ubd84\uc774 \uc190\uc2e4\ud568\uc218\uc640 \uc5ed\uc804\ud30c\uc785\ub2c8\ub2e4. PyTorch\uc5d0\uc11c\ub294 \ub2e4\uc591\ud55c \uc190\uc2e4\ud568\uc218\ub97c \uc81c\uacf5\ud558\ub294\ub370, \uadf8 \uc911 torch.nn.CrossEntropyLoss\ub294 \ub2e4\uc911 \ubd84\ub958\uc5d0 \uc0ac\uc6a9\ub429\ub2c8\ub2e4. torch.nn.CrossEntropyLoss\ub294 nn.LogSoftmax\uc640 nn.NLLLoss\uc758 \uc5f0\uc0b0\uc758 \uc870\ud569\uc785\ub2c8\ub2e4. nn.LogSoftmax\ub294 \uc2e0\uacbd\ub9dd \ub9d0\ub2e8\uc758 \uacb0\uacfc \uac12\ub4e4\uc744 \ud655\ub960\uac1c\ub150\uc73c\ub85c \ud574\uc11d\ud558\uae30 \uc704\ud55c Softmax \ud568\uc218\uc758 \uacb0\uacfc\uc5d0 log \uac12\uc744 \ucde8\ud55c \uc5f0\uc0b0\uc774\uace0, nn.NLLLoss\ub294 nn.LogSoftmax\uc758 log \uacb0\uacfc\uac12\uc5d0 \ub300\ud55c \uad50\ucc28 \uc5d4\ud2b8\ub85c\ud53c \uc190\uc2e4 \uc5f0\uc0b0(Cross Entropy Loss|Error)\uc785\ub2c8\ub2e4.<\/p>\n<p>Softmax\uc640 \uad50\ucc28 \uc5d4\ud2b8\ub85c\ud53c \uc190\uc2e4 \uc5f0\uc0b0\uc5d0 \ub300\ud55c \uac01\uac01\uc758 \uc124\uba85\uc740 \uc544\ub798\uc640 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<blockquote class=\"wp-embedded-content\" data-secret=\"XK1bhV6nAi\"><p><a href=\"http:\/\/www.gisdeveloper.co.kr\/?p=7550\">\ud65c\uc131\ud654 \ud568\uc218(Activation Function)<\/a><\/p><\/blockquote>\n<p><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;\ud65c\uc131\ud654 \ud568\uc218(Activation Function)&#8221; &#8212; GIS Developer\" src=\"http:\/\/www.gisdeveloper.co.kr\/?p=7550&#038;embed=true#?secret=EisV6C9eNY#?secret=XK1bhV6nAi\" data-secret=\"XK1bhV6nAi\" width=\"525\" height=\"296\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe><\/p>\n<blockquote class=\"wp-embedded-content\" data-secret=\"bfm5WZC4RN\"><p><a href=\"http:\/\/www.gisdeveloper.co.kr\/?p=7631\">\uc190\uc2e4\ud568\uc218(Loss Function)<\/a><\/p><\/blockquote>\n<p><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;\uc190\uc2e4\ud568\uc218(Loss Function)&#8221; &#8212; GIS Developer\" src=\"http:\/\/www.gisdeveloper.co.kr\/?p=7631&#038;embed=true#?secret=QWVZQrPIuw#?secret=bfm5WZC4RN\" data-secret=\"bfm5WZC4RN\" width=\"525\" height=\"296\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe><\/p>\n<p>\ucc38\uace0\ub85c nn.NLLLoss\uc758 \uad6c\ud604 \ucf54\ub4dc\ub294 \uc544\ub798\uc640 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nimport torch\r\n\r\ndef NLLLoss(logs, targets):\r\n    out = torch.zeros_like(targets, dtype=torch.float)\r\n    for i in range(len(targets)):\r\n        out[i] = logs[i][targets[i]]\r\n    return -out.sum()\/len(out)\r\n<\/pre>\n<p>\ubb3c\ub860 PyTorch\uc5d0\uc11c\ub3c4 torch.nn.NLLLoss\ub97c \ud1b5\ud574 \uc704\uc640 \ub3d9\uc77c\ud55c \uae30\ub2a5\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4. \uacb0\uacfc\uc801\uc73c\ub85c Softmax\uc758 Log \uacb0\uacfc\ub97c Cross Entropy Loss \uac12\uc758 \uacb0\uacfc\ub97c \uc5bb\uae30 \uc704\ud574 3\uac00\uc9c0 \ubc29\uc2dd\uc774 \uc874\uc7ac\ud558\ub294\ub370, \uc544\ub798\uc640 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nx = torch.Tensor([[0.8982, 0.805, 0.6393, 0.9983, 0.5731, 0.0469, 0.556, 0.1476, 0.8404, 0.5544]])\r\ny = torch.LongTensor([1])\r\n\r\n# Case 1\r\ncross_entropy_loss = torch.nn.CrossEntropyLoss()\r\nprint(cross_entropy_loss(x, y)) # tensor(2.1438)\r\n\r\n# Case 2\r\nlog_softmax = torch.nn.LogSoftmax(dim=1)\r\nx_log = log_softmax(x)\r\nprint(NLLLoss(x_log, y)) # tensor(2.1438)\r\n\r\n# Case 3\r\nnll_loss = torch.nn.NLLLoss()\r\nprint(nll_loss(x_log, y)) # tensor(2.1438)\r\n<\/pre>\n<p>\uc704\uc758 \uc138\uac00\uc9c0 \ubc29\uc2dd \uc911 torch.nn.CrossEntropyLoss\ucc98\ub7fc \uc5f0\uc0b0\uc744 \ud55c\ubc88\uc5d0 \ucc98\ub9ac\ud558\ub294 \uac83\uc774 \uc218\uc2dd\uc774 \uac04\uc18c\ud654\ub418\uc5b4 \uc5ed\uc804\ud30c\uac00 \ub354 \uc548\uc815\uc801\uc73c\ub85c \uc774\ub8e8\uc9c0\ubbc0\ub85c \uc2e4\uc81c \uc0ac\uc6a9\uc5d0 \uad8c\uc7a5\ub429\ub2c8\ub2e4.<\/p>\n<p>torch.nn.CrossEntropyLoss\ub97c \uc774\uc6a9\ud558\uc5ec \uc190\uc2e4\uac12\uc744 \uad6c\ud558\ub294 \uac83\uc5d0 \ucd08\uc810\uc744 \ub9de\ucdb0\ubcf4\uba74.. \uba3c\uc800 torch.nn.CrossEntropyLoss\uc758 \uc218\uc2dd\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<p><center><\/p>\n<p class=\"ql-center-displayed-equation\" style=\"line-height: 60px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/www.gisdeveloper.co.kr\/wp-content\/ql-cache\/quicklatex.com-f3208d5dfaaecd0f28069ec368ceb1f7_l3.png\" height=\"60\" width=\"662\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#36;&#36;&#108;&#111;&#115;&#115;&#40;&#120;&#44;&#99;&#108;&#97;&#115;&#115;&#41;&#61;&#45;&#92;&#108;&#111;&#103;&#92;&#98;&#105;&#103;&#103;&#108;&#40;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#101;&#120;&#112;&#40;&#120;&#91;&#99;&#108;&#97;&#115;&#115;&#93;&#41;&#125;&#123;&#92;&#115;&#117;&#109;&#95;&#123;&#106;&#125;&#123;&#92;&#101;&#120;&#112;&#40;&#120;&#91;&#106;&#93;&#41;&#125;&#125;&#92;&#98;&#105;&#103;&#103;&#114;&#41;&#61;&#45;&#120;&#91;&#99;&#108;&#97;&#115;&#115;&#93;&#43;&#92;&#108;&#111;&#103;&#92;&#98;&#105;&#103;&#103;&#108;&#40;&#92;&#115;&#117;&#109;&#95;&#123;&#106;&#125;&#123;&#92;&#101;&#120;&#112;&#40;&#120;&#91;&#106;&#93;&#41;&#125;&#125;&#92;&#98;&#105;&#103;&#103;&#114;&#41;&#36;&#36;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>\n<p><\/center><\/p>\n<p>\uc704\uc758 \uc218\uc2dd\uc744 \uc0b4\ud3b4\ubcf4\uba74 \uc55e\uc11c \uc5b8\uae09\ud55c \uac83\ucc98\ub7fc Softmax\uc640 Log\ucc98\ub9ac \ubc0f Cross Entropy Loss \uc5f0\uc0b0\uc758 \uc870\ud569\uc774\ub77c\ub294 \uac83\uc744 \uc54c\uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n<p>torch.nn.CrossEntropyLoss\ub97c \ucf54\ub4dc\ub97c \ud1b5\ud574 \uc124\uba85\ud558\uba74&#8230; \uc608\ub97c\ub4e4\uc5b4 \uc2e0\uacbd\ub9dd \ub9d0\ub2e8\uc5d0\uc11c \ucd1d 10\uac1c\uc758 \uac12(\uc55e\uc11c \uc5b8\uae09\ud55c x\uac12)\uc774 \ucd9c\ub825\ub418\uc5c8\uace0, \uc2e4\uc81c \ub808\uc774\ube14 \uac12(\uc55e\uc11c \uc5b8\uae09\ud55c y \ub610\ub294 class)\uc740 1\uc77c\ub54c\uc5d0 \uc190\uc2e4\uac12\uc744 \uad6c\ud558\ub294 \ucf54\ub4dc\ub294 \uc544\ub798\uc640 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nimport torch\r\nimport torch.nn as nn\r\nimport numpy as np\r\n\r\noutput = torch.Tensor([[0.8982, 0.805, 0.6393, 0.9983, 0.5731, 0.0469, 0.556, 0.1476, 0.8404, 0.5544]])\r\ntarget = torch.LongTensor([1])\r\n\r\ncriterion = nn.CrossEntropyLoss()\r\nloss = criterion(output, target)\r\nprint(loss) # tensor(2.1438)\r\n<\/pre>\n<p>\ucc38\uace0\ub85c \uc704\uc758 \ucf54\ub4dc\uc5d0\uc11c \uc0ac\uc6a9\ub41c nn.CrossEntropyLoss\uc758 \uc218\uc2dd\uc744 \uc54c\uace0 \uc788\uc73c\ubbc0\ub85c nn.CrossEntropyLoss\uc744 \uc0ac\uc6a9\ud558\uc9c0 \uc54a\uace0 \uc9c1\uc811 \uc190\uc2e4\uac12\uc744 \uacc4\uc0b0\ud55c\ub2e4\uba74 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nimport torch\r\nimport torch.nn as nn\r\nimport numpy as np\r\n\r\noutput = [0.8982, 0.805, 0.6393, 0.9983, 0.5731, 0.0469, 0.556, 0.1476, 0.8404, 0.5544]\r\ntarget = [1]\r\nloss = np.log(sum(np.exp(output))) - output[target[0]]\r\nprint(loss) # 2.143818427948945\r\n<\/pre>\n<p>\uc190\uc2e4\uac12\uc774 \ud544\uc694\ud560 \ub54c\ub294 \uc2e0\uacbd\ub9dd\uc758 \ud559\uc2b5\uc778\ub370, \ud559\uc2b5\uc5d0\uc11c \ub370\uc774\ud130\ub294 GPU \uc790\uc6d0\uc744 \ucd5c\ub300\ud55c \ud65c\uc6a9\ud558\uae30 \uc704\ud574 \ubc30\uce58 \ub2e8\uc704\ub85c \ucc98\ub9ac\ub429\ub2c8\ub2e4. \uc989, \uc55e\uc11c \uc5b8\uae09\ud55c \uac83\ucc98\ub7fc 1\uac1c \ub2e8\uc704\uac00 \uc544\ub2cc 2\uac1c \uc774\uc0c1\uc758 \ub370\uc774\ud130\uac00 \ud55c\uaebc\ubc88\uc5d0 \ub4e4\uc5b4\uc628\ub2e4\ub294 \uac83\uc785\ub2c8\ub2e4. \uc774\uc5d0 \ub300\ud55c \ucc98\ub9ac\uc5d0 \ub300\ud55c \uc608\ub294 \ub2e4\uc74c \ucf54\ub4dc\uc640 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nimport torch\r\nimport torch.nn as nn\r\nimport numpy as np\r\n\r\noutput = torch.Tensor(\r\n    [\r\n        [0.8982, 0.805, 0.6393, 0.9983, 0.5731, 0.0469, 0.556, 0.1476, 0.8404, 0.5544],\r\n        [0.9457, 0.0195, 0.9846, 0.3231, 0.1605, 0.3143, 0.9508, 0.2762, 0.7276, 0.4332]\r\n    ]\r\n)\r\n\r\ntarget = torch.LongTensor([1, 5])\r\n\r\ncriterion = nn.CrossEntropyLoss()\r\nloss = criterion(output, target)\r\nprint(loss) # tensor(2.3519)\r\n<\/pre>\n<p>\uc704\uc758 \ucf54\ub4dc\ub97c nn.CrossEntropyLoss()\ub97c \uc0ac\uc6a9\ud558\uc9c0 \uc54a\uace0 \uacc4\uc0b0\ud55c\ub2e4\uba74 \ub2e4\uc74c \ucf54\ub4dc\uc640 \uac19\uad6c\uc694.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">\r\nimport torch\r\nimport torch.nn as nn\r\nimport numpy as np\r\n\r\noutput = [0.8982, 0.805, 0.6393, 0.9983, 0.5731, 0.0469, 0.556, 0.1476, 0.8404, 0.5544]\r\ntarget = [1]\r\nloss1 = np.log(sum(np.exp(output))) - output[target[0]]\r\n\r\noutput = [0.9457, 0.0195, 0.9846, 0.3231, 0.1605, 0.3143, 0.9508, 0.2762, 0.7276, 0.4332]\r\ntarget = [5]\r\nloss2 = np.log(sum(np.exp(output))) - output[target[0]]\r\n\r\nprint((loss1 + loss2)\/2) # 2.351937720511233\r\n<\/pre>\n<p>\uc989, \ubc30\uce58 \ucc98\ub9ac\uc5d0 \ub300\ud55c \uc190\uc2e4\uac12\uc740 \ubc30\uce58\ub97c \uad6c\uc131\ud558\ub294 \uac01 \ub370\uc774\ud130\uc758 \uc190\uc2e4\uac12\ub4e4\uc758 \ud3c9\uade0\uc774\ub77c\ub294 \uc810\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\ub525\ub7ec\ub2dd\uc758 \ub9ce\uc740 \uc774\ub860 \uc911 \uac00\uc7a5 \uc911\uc694\ud55c \ubd80\ubd84\uc774 \uc190\uc2e4\ud568\uc218\uc640 \uc5ed\uc804\ud30c\uc785\ub2c8\ub2e4. PyTorch\uc5d0\uc11c\ub294 \ub2e4\uc591\ud55c \uc190\uc2e4\ud568\uc218\ub97c \uc81c\uacf5\ud558\ub294\ub370, \uadf8 \uc911 torch.nn.CrossEntropyLoss\ub294 \ub2e4\uc911 \ubd84\ub958\uc5d0 \uc0ac\uc6a9\ub429\ub2c8\ub2e4. torch.nn.CrossEntropyLoss\ub294 nn.LogSoftmax\uc640 nn.NLLLoss\uc758 \uc5f0\uc0b0\uc758 \uc870\ud569\uc785\ub2c8\ub2e4. nn.LogSoftmax\ub294 \uc2e0\uacbd\ub9dd \ub9d0\ub2e8\uc758 \uacb0\uacfc \uac12\ub4e4\uc744 \ud655\ub960\uac1c\ub150\uc73c\ub85c \ud574\uc11d\ud558\uae30 \uc704\ud55c Softmax \ud568\uc218\uc758 \uacb0\uacfc\uc5d0 log \uac12\uc744 \ucde8\ud55c \uc5f0\uc0b0\uc774\uace0, nn.NLLLoss\ub294 nn.LogSoftmax\uc758 log \uacb0\uacfc\uac12\uc5d0 \ub300\ud55c \uad50\ucc28 \uc5d4\ud2b8\ub85c\ud53c \uc190\uc2e4 \uc5f0\uc0b0(Cross Entropy Loss|Error)\uc785\ub2c8\ub2e4. Softmax\uc640 \uad50\ucc28 \uc5d4\ud2b8\ub85c\ud53c \uc190\uc2e4 \uc5f0\uc0b0\uc5d0 &hellip; <\/p>\n<p class=\"link-more\"><a href=\"http:\/\/www.gisdeveloper.co.kr\/?p=8668\" class=\"more-link\">\ub354 \ubcf4\uae30<span class=\"screen-reader-text\"> &#8220;\ub2e4\uc911\ubd84\ub958\ub97c \uc704\ud55c \ub300\ud45c\uc801\uc778 \uc190\uc2e4\ud568\uc218, torch.nn.CrossEntropyLoss&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[131,132,1],"tags":[],"class_list":["post-8668","post","type-post","status-publish","format-standard","hentry","category-python","category-deep-machine-learning","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/8668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8668"}],"version-history":[{"count":14,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/8668\/revisions"}],"predecessor-version":[{"id":8672,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/8668\/revisions\/8672"}],"wp:attachment":[{"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8668"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.gisdeveloper.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}