Tensorflow.JS + React.JS – requestAnimationFrame with nothing being drawn onto the canvas

Building a lightweight React app that leverages tfjs to make real time detections using a custom model. The app works fine when looping using setInterval but it's a little glitchy when rendering.

I'm currently performance tuning in hopes to get better fps and less 'glitching' when rendering animations to the canvas, everything I've read points to using requestAnimationFrame in order to smooth out the rendering however right now it's not actually drawing anything to the canvas. console.log'in out results from the drawRect function are successful but it's not actually rendering anything to the canvas.

TL;DR;

  • Detections from tfjs model not being rendered to canvas
  • Works when using setInterval, doesn't when using requestAnimation to loop
  /// App.JS
  const webcamRef = useRef(null);
  const canvasRef = useRef(null);

  const runCoco = async () => {
    const net = await tf.loadGraphModel('JSON HOSTED MODEL')
    requestAnimationFrame(()=>detect(net))
    
  };

  const detect = async (net) => {
    
    // Check data is available
    if (
      typeof webcamRef.current !== "undefined" &&
      webcamRef.current !== null &&
      webcamRef.current.video.readyState === 4
    ) {
      // Get Video Properties
      const video = webcamRef.current.video;
      const videoWidth = webcamRef.current.video.videoWidth;
      const videoHeight = webcamRef.current.video.videoHeight;

      // Set video width
      webcamRef.current.video.width = videoWidth;
      webcamRef.current.video.height = videoHeight;

      // Set canvas height and width
      canvasRef.current.width = videoWidth;
      canvasRef.current.height = videoHeight;

      const img = tf.browser.fromPixels(video)
      const resized = tf.image.resizeBilinear(img, [640,480])
      const casted = resized.cast('int32')
      const expanded = casted.expandDims(0)
      const obj = await net.executeAsync(expanded)

      /// Extract components from detections
      const boxes = await obj[1].array()  
      const classes = await obj[2].array()
      const scores = await obj[4].array()
            
      // Draw boxes
      const ctx = canvasRef.current.getContext("2d");

      ////// THIS BIT ISN'T ACTUALLY DRAWING //////// halppls
      drawRect(boxes[0], classes[0], scores[0], 0.01, videoWidth, videoHeight, ctx);

      // Cleanup
      tf.dispose(img)
      tf.dispose(resized)
      tf.dispose(casted)
      tf.dispose(expanded)
      tf.dispose(obj)
  
    }
    requestAnimationFrame(()=>detect(net))
  };

  useEffect(()=>{runCoco()},[]);

This is the drawing function that's being used:

export const drawRect = (boxes, classes, scores, threshold, imgWidth, imgHeight, ctx) =>{
  for(let i=0; i <= boxes.length; i++){
    if(boxes[i] && classes[i] && scores[i]>threshold){
      const [y, x, height, width] = boxes[i]; 
      const text = classes[i];
      const color = 'red'
      ctx.beginPath();
      ctx.strokeStyle = color
      ctx.font = '18px Arial';
      ctx.fillStyle = '#' + color
      ctx.fillText(text, x*imgWidth, y*imgHeight);
      ctx.rect(x*imgWidth, y*imgHeight, width*imgWidth/1.5, height*imgHeight/1.2);    
      ctx.stroke();  
  }
  }
}


Read more here: https://stackoverflow.com/questions/64938976/tensorflow-js-react-js-requestanimationframe-with-nothing-being-drawn-onto-t

Content Attribution

This content was originally published by nicknochnack at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: