タイトル | : SINGLE型の切捨てについて |
記事No | : 11755 |
投稿日 | : 2016/11/08(Tue) 16:43 |
投稿者 | : SUZUKI |
また壁にぶつかってしまいました 下記はSINGLE型にて小数点N位以下を切捨てするロジックです (今回は小数点第4位以下を切り捨て) 型を色々変えていますがうまく行っていません DOUBLE型は色々検索に出て来るのですが SINGLE型はうまくヒットしませんでした? Math.Floor関数がDOUBLE型なので変換誤差をうまく吸収できずにいます Math.Floor関数あきらめたほうがよいか教えていただきたく よろしくお願いします
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click Dim a As Single Dim b As Single a = 1.00401 b = CSng(csp_double_syousuu(CDbl(a), 3, 0)) MsgBox(b) ' 1.004 OK b = CSng(csp_single_syousuu(a, 3, 0)) MsgBox(b) ' 1.004 OK b = CSng(csp_decimal_syousuu(CDec(a), 3, 0)) MsgBox(b) ' 1.004 OK a = 1.004 b = CSng(csp_double_syousuu(CDbl(a), 3, 0)) MsgBox(b) '1.003 NO b = CSng(csp_single_syousuu(a, 3, 0)) MsgBox(b) ' 1.003 NO b = CSng(csp_decimal_syousuu(CDec(a), 3, 0)) MsgBox(b) ' 1.004 OK a = 1.00301 b = CSng(csp_double_syousuu(CDbl(a), 3, 0)) MsgBox(b) ' 1.003 OK b = CSng(csp_single_syousuu(a, 3, 0)) MsgBox(b) ' 1.003 OK b = CSng(csp_decimal_syousuu(CDec(a), 3, 0)) MsgBox(b) ' 1.003 OK a = 1.003 b = CSng(csp_double_syousuu(CDbl(a), 3, 0)) MsgBox(b) '1.003 OK b = CSng(csp_single_syousuu(a, 3, 0)) MsgBox(b) ' 1.003 OK b = CSng(csp_decimal_syousuu(CDec(a), 3, 0)) MsgBox(b) ' 1.002 NO
End Sub
''' 見ずらく申し訳ありませんがdValueの型を色々変えています Function csp_single_syousuu(ByRef dValue As Single, ByRef iDigits As Integer, ByRef isw As Integer) As Double Dim dCoef As Double = System.Math.Pow(10, iDigits) Select Case isw Case 0 Return System.Math.Floor(dValue * dCoef) / dCoef End Select End Function
Function csp_double_syousuu(ByRef dValue As Double, ByRef iDigits As Integer, ByRef isw As Integer) As Double Dim dCoef As Double = System.Math.Pow(10, iDigits) Select Case isw Case 0 Return System.Math.Floor(dValue * dCoef) / dCoef End Select End Function
Function csp_decimal_syousuu(ByRef dValue As Decimal, ByRef iDigits As Integer, ByRef isw As Integer) As Double Dim dCoef As Double = System.Math.Pow(10, iDigits) Select Case isw Case 0 Return System.Math.Floor(dValue * dCoef) / dCoef End Select End Function
|