IIO: bmp280-core.c: fix error in humidity calculation

PATCH Applied Andreas Klinger 4 participants 4 messages Apr 10, 2017 – Apr 14, 2017 IIO

Share & tools

Andreas Klinger

author

2017-04-10 17:00 UTC #0 lore raw

PATCH v2 1/1 IIO: bmp280-core.c: fix error in humidity calculation

While calculating the compensation of the humidity there are negative values interpreted as unsigned because of unsigned variables used. These values as well as the constants need to be casted to signed as indicated by the documentation of the sensor.

Changes in v2:
- cast also constants to signed; otherwise calculation is not correct

I removed the Acked-By of Linus Walleij because of another change in the calculation. Don't know how it should be treated.

Signed-off-by Andreas Klinger <[email protected]>

---
 drivers/iio/pressure/bmp280-core.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

drivers/iio/pressure/bmp280-core.c +6 -5

index 4d18826ac63c..d82b788374b6 100644
@@ -175,11 +175,12 @@ static u32 bmp280_compensate_humidity(struct bmp280_data *data,
 	}
 	H6 = sign_extend32(tmp, 7);
 
-	var = ((s32)data->t_fine) - 76800;
-	var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var)) + 16384) >> 15)
-		* (((((((var * H6) >> 10) * (((var * H3) >> 11) + 32768)) >> 10)
-		+ 2097152) * H2 + 8192) >> 14);
-	var -= ((((var >> 15) * (var >> 15)) >> 7) * H1) >> 4;
+	var = ((s32)data->t_fine) - (s32)76800;
+	var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var))
+		+ (s32)16384) >> 15) * (((((((var * H6) >> 10)
+		* (((var * (s32)H3) >> 11) + (s32)32768)) >> 10)
+		+ (s32)2097152) * H2 + 8192) >> 14);
+	var -= ((((var >> 15) * (var >> 15)) >> 7) * (s32)H1) >> 4;
 
 	return var >> 12;
 };

Signature · 4 lines

2.1.4


--

Linus Walleij

Acked-by on v2 1/1

2017-04-11 07:45 UTC #1 lore raw

Quoted 10 lines · Andreas Klinger

While calculating the compensation of the humidity there are negative values interpreted as unsigned because of unsigned variables used. These values as well as the constants need to be casted to signed as indicated by the documentation of the sensor.
Changes in v2:
- cast also constants to signed; otherwise calculation is not correct
I removed the Acked-By of Linus Walleij because of another change in the calculation. Don't know how it should be treated.

Acked-by Linus Walleij <[email protected]>

You guys do an awesome job at maintaining the sensor code, thanks.

Yours,
Linus Walleij

Matt Ranostay

Reviewed-by on v2 1/1

2017-04-11 23:59 UTC #2 lore raw

In reply to Andreas Klinger

Quoted 5 lines · Andreas Klinger

While calculating the compensation of the humidity there are negative values interpreted as unsigned because of unsigned variables used. These values as well as the constants need to be casted to signed as indicated by the documentation of the sensor.

Quoted 7 lines

Changes in v2:
- cast also constants to signed; otherwise calculation is not correct
I removed the Acked-By of Linus Walleij because of another change in the calculation. Don't know how it should be treated.
Signed-off-by Andreas Klinger <[email protected]>

Reviewed-by Matt Ranostay <[email protected]>

Quoted 31 lines

---
 drivers/iio/pressure/bmp280-core.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

drivers/iio/pressure/bmp280-core.c +6 -6

index 4d18826ac63c..d82b788374b6 100644
@@ -175,11 +175,12 @@ static u32 bmp280_compensate_humidity(struct bmp280_data *data,
        }
        H6 = sign_extend32(tmp, 7);
 
-       var = ((s32)data->t_fine) - 76800;
-       var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var)) + 16384) >> 15)
-               * (((((((var * H6) >> 10) * (((var * H3) >> 11) + 32768)) >> 10)
-               + 2097152) * H2 + 8192) >> 14);
-       var -= ((((var >> 15) * (var >> 15)) >> 7) * H1) >> 4;
+       var = ((s32)data->t_fine) - (s32)76800;
+       var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var))
+               + (s32)16384) >> 15) * (((((((var * H6) >> 10)
+               * (((var * (s32)H3) >> 11) + (s32)32768)) >> 10)
+               + (s32)2097152) * H2 + 8192) >> 14);
+       var -= ((((var >> 15) * (var >> 15)) >> 7) * (s32)H1) >> 4;
 
        return var >> 12;
 };
--

2.1.4

Jonathan Cameron

maintainer Applied

2017-04-14 14:13 UTC #3 lore raw

Quoted 9 lines · Matt Ranostay

Quoted 8 lines · Andreas Klinger
While calculating the compensation of the humidity there are negative values interpreted as unsigned because of unsigned variables used. These values as well as the constants need to be casted to signed as indicated by the documentation of the sensor.
Changes in v2:
- cast also constants to signed; otherwise calculation is not correct

Quoted 6 lines

Quoted 4 lines
I removed the Acked-By of Linus Walleij because of another change in the calculation. Don't know how it should be treated.
Signed-off-by Andreas Klinger <[email protected]>
Reviewed-by Matt Ranostay <[email protected]>

Applied to the fixes-togreg branch of iio.git and marked for stable.

Thanks,

Jonathan

Quoted 32 lines

---
 drivers/iio/pressure/bmp280-core.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

drivers/iio/pressure/bmp280-core.c +6 -6

index 4d18826ac63c..d82b788374b6 100644
@@ -175,11 +175,12 @@ static u32 bmp280_compensate_humidity(struct bmp280_data *data,
        }
        H6 = sign_extend32(tmp, 7);
 
-       var = ((s32)data->t_fine) - 76800;
-       var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var)) + 16384) >> 15)
-               * (((((((var * H6) >> 10) * (((var * H3) >> 11) + 32768)) >> 10)
-               + 2097152) * H2 + 8192) >> 14);
-       var -= ((((var >> 15) * (var >> 15)) >> 7) * H1) >> 4;
+       var = ((s32)data->t_fine) - (s32)76800;
+       var = ((((adc_humidity << 14) - (H4 << 20) - (H5 * var))
+               + (s32)16384) >> 15) * (((((((var * H6) >> 10)
+               * (((var * (s32)H3) >> 11) + (s32)32768)) >> 10)
+               + (s32)2097152) * H2 + 8192) >> 14);
+       var -= ((((var >> 15) * (var >> 15)) >> 7) * (s32)H1) >> 4;
 
        return var >> 12;
 };
--

2.1.4