Invalid argument provided to Gemini: 400 Please ensure that function call turn comes immediately after a user turn or after a function response turn.

[Divyanshu9822]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.schema.messages import SystemMessage
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain.agents import AgentExecutor
from langchain.agents import create_tool_calling_agent
from assistant.memory import memory


class Assistant:
    def __init__(self, llm, tools):
        self.agent = self._create_inference_chain(llm, tools)

    def answer(self, question: str, user_id: str) -> str:
        """
        Process a user's question and generate a response.

        :param question: The user's input question.
        :param user_id: The unique identifier for the user.
        :return: The AI-generated response to the question.
        """
        if not question:
            return

        previous_memories = memory.search(question, user_id=user_id, limit=3)

        relevant_memories_text = "\n".join(
            mem["memory"] for mem in previous_memories["results"]
        )

        prompt = f"""
        User input: {question}

        Relevant memories: 
        
        {relevant_memories_text}
        """
        response = self.agent.invoke(
            {"prompt": prompt},
            config={"configurable": {"session_id": "unused"}},
        )

        return response["output"]

    def _create_inference_chain(self, llm, tools):
        SYSTEM_PROMPT = """
        You are an AI personal assistant with context awareness, long-term memory, ability to take and interpret screenshots using along with capturing images using device camera. Your job is to assist the user, handle queries regarding the screen and the image clicked using camera, remember key details from conversations, and provide personalized support. Use past interactions to adapt responses and make future conversations more efficient. Respond naturally like a human, without explaining the reasoning behind your responses or why you chose them.
        """

        prompt_template = ChatPromptTemplate.from_messages(
            [
                SystemMessage(content=SYSTEM_PROMPT),
                MessagesPlaceholder(variable_name="chat_history", n_messages=3),
                (
                    "human",
                    [
                        {"type": "text", "text": "{prompt}"},
                    ],
                ),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
        )

        agent = create_tool_calling_agent(llm, tools, prompt=prompt_template)
        agent_executor = AgentExecutor(agent=agent, tools=tools)

        chat_message_history = ChatMessageHistory()
        return RunnableWithMessageHistory(
            agent_executor,
            lambda _: chat_message_history,
            input_messages_key="prompt",
            history_messages_key="chat_history",
        )

Description

Please help me with this issue, I get it randomly. As far i understood there is some sequencing issue but i don't understand how could it be and how to even fix. It's been days without solution. I'm giving the code below as well for reference. Any guidance/solution is appreciated.

Edit : I've added the verbose=True in AgentExecutor and got the logs where i can see that the assistant was able to recognise me correctly but still somehow couldn't gave the response and thrown the error langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 Please ensure that function call turn comes immediately after a user turn or after a function response turn.

Screenshot (debug)

System Info

Package Information

langchain_core: 0.2.41
langchain: 0.2.16
langchain_community: 0.2.17
langsmith: 0.1.129
langchain_google_genai: 1.0.10
langchain_text_splitters: 0.2.4

[dosubot[bot]]

Hello, @Divyanshu9822! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer.

I found a similar open issue titled "Tool calling with Gemini through Vertex AI fails because of argument with Union type" which might be related to your problem. This issue is still open and was last updated on September 06, 2024 ^[1].

The error message you're encountering suggests that there might be a sequencing issue with the function call turn in your code. Specifically, the function call should come immediately after a user turn or a function response turn. This means that the order of messages in your chat history might not be correctly maintained, leading to the error.

Here are a few things you can check or try to resolve the issue:

1. Message Sequencing: Ensure that the sequence of messages in your ChatMessageHistory is correct. The function call should follow a user message or a function response. You might need to review how messages are added to the history and ensure they are in the correct order.

2. Configuration: Verify that the configurable parameter in your AgentExecutor is correctly set up. The configurable parameter should map context configuration IDs to callable functions, which are used during message processing. Ensure that the session ID or any other context-specific configurations are correctly implemented.

3. Prompt Template: Check the ChatPromptTemplate to ensure that the placeholders are correctly defined and that the messages are being inserted in the correct order. The MessagesPlaceholder should be used appropriately to maintain the sequence of messages.

4. Debugging: Add logging or print statements to track the flow of messages and see where the sequence might be breaking. This can help identify if there are any unexpected message orders or missing messages.

By ensuring that the message sequence is correctly maintained and that the configuration is properly set up, you should be able to resolve the issue. If the problem persists, reviewing the related open issue might provide additional insights or updates on a potential fix.

To continue talking to Dosu, mention @dosu.

[Divyanshu9822]

@dosu I have added verbose = True in AgentExecutor and it gave logs with correct responses while the flow was inside the chain but thrown error after ending chain when flow when to AI agent again to output the response

[dosubot[bot]]

The error you're encountering might be related to how the transition is handled between the end of the chain and returning control to the AI agent. In the LangChain framework, the AgentExecutor uses the AgentExecutorIterator class to manage this transition. When the final output is reached, the _process_next_step_output method checks if the output is an instance of AgentFinish. If it is, the _return method is called to handle the final output logic, which includes processing the final output, updating the run manager, and returning the final outputs ^[1].

Given that the error occurs after the chain ends, it might be worth checking how the final outputs are being handled and returned to the AI agent. Ensure that the sequence of messages is correctly maintained and that the final output logic is properly implemented. You might also want to review the configuration of the AgentExecutor to ensure that it aligns with the expected message flow and error handling.

To continue talking to Dosu, mention @dosu.

[Divyanshu9822]

I've resolved this myself, after tracking it properly and storing every state and comparing i figured out that due to the param n_messages=5 previous messages gets cleared due to which any type of message like HumanMessage, AIMessage and ToolMessage can become first but gemini needs messages to be started with HumanMessage and for tool calling we need fucntion call messages as well. So because of all these i think there was the error and now everything working fine without that params.